Andy,
These folks
http://investbyagent.com/
presented their DeepGreen automated trading system at codecon last year. If their order execution infrastructure is declassified, perhaps you can find some reference to it on the web or by contacting them directly. Though not open source, presenting at codecon seems to imply a certain openness ...
My sense is that the *ML boat hasn't yet arrived in the financial world. I'm sure what you want exists as a packaged API/service just like the various e-commerce offerings, but expect to shell out some dough.