MOE

To be fair, the hyperparameter tuning behind these AutoML systems are getting fairly robust. Google bases theirs on Vizier [0]. The Amazon Sagemaker group has people from the gpyopt project [1]. There is also tons of open source projects out there to help for non-enterprise projects [2] [3]. There are also stand-alone companies that help with this explicitly for enterprises [4] (Caveat, I am a founder). Increasingly I think more time will be spent on the creative/bespoke aspects you mention later in your post, like making sure that you are building a system that actually achieves some business value (vs just getting a better academic-oriented metric result). Hyperparameter tuning is basically trying to do high-dimensional, non-convex optimization on time consuming and expensive to sample functions. Hand tuning is a terrible way to approac

SigOpt (YC W15) is a related service that performs a superset of these features as a SaaS API (I am one of the co-founders). We've been solving this problem for customers around the world for the last 4 years and have extended a lot of the original research I started at Yelp with MOE [1]. We employ an ensemble of optimization techniques and provide seamless integration with any pipeline. I'm happy to answer any questions about the product or technology. We're completely free for academics [2] and publish research like the above at ICML, NIPS, AISTATS regularly [3]. [1]: https://github.com/yelp/MOE [2]: https://sigopt.com/edu [3]: https://sigopt.com/research

It is referenced in the paper, but it's worth pointing out Yelp's MOE. This is an open source (Apache 2.0) implementation for black-box optimization. It doesn't look like it is well maintained, but it is interesting nonetheless. https://github.com/Yelp/MOE

Yes, but does it work? I cannot tell from reading the introductory materials whether it will actually search efficiently over large space (where it is computationally expensive to evaluate the fitness of any individual point). I know that folks are having some success with Gaussian process optimisation for hyper parameter tuning (eg https://github.com/Yelp/MOE), but I would be sceptical about the application of genetic programming to this area, particularly as broadly defined a search space as TPOT seems to set up (genetic programming either doesn't make assumptions about the objective function being optimised which can be used to search more efficiently, like Gaussian optimisation does - or uses implicit assumptions that may be unsuitable, depending on how you want to think about it); has anyone seen benchmarks against other search metho

As you pointed out, it is all about a balance, and every feature has different tradeoffs. SigOpt was designed to unlock the power of Bayesian optimization for anyone doing machine learning. We believe that you shouldn't need to be an expert and spend countless hours of administration to achieve great results for every model. We're wrapping an ensemble of the best Bayesian methods behind a simple interface [0] and constantly making improvements so that people can focus on designing features and their individual domain expertise, instead of needing to build and maintain their own hyperparameter optimization tools to see the benefit. For experts who want to spend a lot of time and effort customizing, administering, updating, and maintaining a hyperparameter tuning solution I would recommend forking one of the open source packages out there l

We've found that SigOpt compares very well to spearmint, as well as MOE [1], which I wrote and open sourced around the same time spearmint was open sourced. We have a paper coming out soon comparing SigOpt rigorously to standard methods like random and grid search as well as other open source Bayesian methods like MOE [1], spearmint, HyperOpt [2], and SMAC [3] with good results. [1]: https://github.com/Yelp/MOE [2]: https://github.com/hyperopt/hyperopt [3]: http://www.cs.ubc.ca/labs/beta/Projects/SMAC/

Depending on how long it takes to "evaluate the goodness of a solution" techniques like multi-start gradient decent can rapidly become intractable though, especially in higher dimensions. There are a handful of open source libraries out there that try to tackle this time consuming and expensive black box optimization problem from a more Bayesian approach [1] [2]. There's also a YC company to do this as a service (full disclosure, I'm a co-founder) [3]. Combining these methods with TensorFlow could be very powerful and is one step closer to more automatic ML flows though. [1]: https://github.com/Yelp/MOE (full disclosure, I co-wrote this) [2]: https://github.com/hyperopt/hyperopt [3]: https://sigopt.com/cases/machine_learning

Thanks for the feedback! We'll update the post to be more clear about this. Our goal was to make this as simple and accessible as possible and in doing so we may have cut out too much. I'll try to make this more clear. SigOpt [0] vs OSS: We are similar to hyperopt, spearmint, and MOE [1], which we developed at Yelp and also uses Gaussian Processes to do Bayesian Global Optimization. SigOpt extends and expands upon our work on MOE while wrapping everything in a simple API [2] and web interface. One thing we learned while promoting MOE was that many people have this problem, but few have the time or expertise to get these expert level open source tools running properly, so we built SigOpt to bring these powerful tools to anyone via a simple API. [0]: https://sigopt.com [1]: https://github.com/Yelp/MOE [2]: https://sigopt.com/docs

There are a variety of different techniques that solve similar problems. We have some of them documented here: http://yelp.github.io/MOE/why_moe.html#other-methods (including these two), but would love to add more. Each one has different tradeoffs depending on what your problem entails, what license you can use and what platform you are running on. Many of these methods have been around for some time. The seminal paper was published in 1998: http://www.ressources-actuarielles.net/EXT/ISFA/1226.nsf/0/f... MOE provides many interfaces (REST, Python, C++), a permissive license (Apache v2) and allows for many optimal points to be quickly queried at once, non-sequentially (Expected Parallel Improvement), along with a slew of other features. We have also tried to follow strict software engineering principles throughout (testing, docing, etc). W