sigopt-examples

Post tl;dr: Sometimes when you are optimizing an ML / AI pipeline you care about multiple things (speed, accuracy, etc). It is often difficult to make these tradeoffs without knowing what is possible, but it is expensive to try different things. Using Bayesian optimization to optimize multiple metrics simultaneously can help solve this problem more than an order of magnitude faster than standard techniques like randomized search. We provide examples tuning TensorFlow and MXnet CNNs with code [1] [2]. SigOpt tl;dr: Parameter optimization-as-a-service. An ensemble of Bayesian optimization techniques behind a simple REST API. Spend more time building your models while we optimally tune them exponentially more efficiently than something like a brute force grid search. Free for academic use [3]. [1]: https://github.com/sigopt/sigopt-examples/t

I'm one of the co-authors of this post and co-founders of SigOpt (YC W15). I'm happy to answer any questions about the post. You can find the code to replicate everything here [1]. More info on the research behind SigOpt can be found here [2]. [1]: https://github.com/sigopt/sigopt-examples/tree/master/dnn-tu... [2]: https://sigopt.com/research

We compare to some standard convex optimization techniques implemented in scipy here [1]. We have comparisons to some other Bayesian methods here [2]. I'm happy to answer any questions! [1]: https://github.com/sigopt/sigopt-examples/blob/master/ipytho... [2]: http://arxiv.org/abs/1603.09441

The methods in scipy.optimize are great if your function is serial, cheap, deterministic, and convex. Many ML models and real world problems don't fit into this context though. We have a Jupyter notebook showing how SigOpt compares to several scipy.optimize methods as well as standard methods like grid/random on a simple non-convex problem here [1]. These results only get more striking as the dimensionality increases. [1]: https://github.com/sigopt/sigopt-examples/blob/master/ipytho...

> ALS implementation in MLLib still requires ratings in RDD and hasn't moved over to ML/DataFrames yet. ALS has been available in ML/DataFrames since 1.3.0, per the documentation. (http://spark.apache.org/docs/latest/api/scala/index.html#org...) > Edit: Looks like the original comment was edited, but this post does in fact use the built in MLlib cross validation methods. I edited the comment to correct my error that ALS was not mentioned as being native to Spark. However, for Hyperparameter cross validation, I looked at the code in the repository (https://github.com/sigopt/sigopt-examples/blob/master/spark/...), and while the Spark CrossValidation function is imported at the beginning of the Scala file (from ML, not MLlib), it is not used in the code, in favor of SigOpt. I'm confused.

Hi, I'm one of the co-founders of SigOpt (YC W15) and am happy to answer any questions about this post or anything about the methods used. More info on the Bayesian optimization behind SigOpt can be found on our research page [1]. The code used in this post can be found on github [2]. [1]: https://sigopt.com/research [2]: https://github.com/sigopt/sigopt-examples/tree/master/dnn-tu...

Thanks for the note! We're using Bayesian optimization to tune both the hyperparameters of the unsupervised model and the supervised model, but you are correct that they are being done in unison with the overall accuracy being the target. The lift you get from adding the unsupervised step (and tuning it) is quite substantial (and statistically significant). The idea of tuning just the unsupervised part (or doing it independently) is great though. All the code for the post is available at https://github.com/sigopt/sigopt-examples/tree/master/unsupe.... It would be interesting to see if doing that would make for a better overall accuracy.

Great idea! You could reuse most of the code [1] to give this a shot. In future posts we may extend this to other sports as well. [1]: https://github.com/sigopt/sigopt-examples

We give a high level overview in footnote #4 of the post, but more detail can be found in the code [1]. We tried to pick a relatively small set of features (and kept those picks constant) in order to isolate the model tuning gains from pure feature selection. You can fork the code and try any other interesting features you think would make it better (there is a ton of data we didn't have the chance to look at). [1]: https://github.com/sigopt/sigopt-examples/blob/master/sigopt...

Post author and co-founder of SigOpt (YC W15) here. Thanks for all the great questions and comments, I'll be around all day answering questions. Feel free to ask anything about the post or what we do at SigOpt (or how). All of the code can be found here: https://github.com/sigopt/sigopt-examples More about how SigOpt works here: https://sigopt.com/research Other discussions here: https://www.reddit.com/r/MachineLearning/comments/3yy0vp/usi... and here: https://news.ycombinator.com/item?id=10819170

Thanks! Great question. There is a huge variance in the cumulative gains as you pointed out. Because of the 100 win/110 loss edge that Vegas has you need to win over 52.5% of the time to register a profit. Even with a model that can win 52.5% + e of the time you are effectively flipping a (biased) coin. Over the 131 games in the holdout dataset this random walk will tend upward, but can vary widely (especially at the start). As the season continues on this should start to wash out (in the cumulative profit). We plan to write a followup post at the end of the season to see how it performs. You could also fork the code [1] and train/tune on '00-'13/'13-'14 and use the entire '14-'15 set as the holdout (or a similar combination). The features can be found here [2], we don't use individual player stats or temporal stats, but it should be rela

Hello, I'm the author of this post and co-founder of SigOpt (YC W15). Let me know if you have any questions about the post or what we do at SigOpt. All of the code used in this post can be found at https://github.com/sigopt/sigopt-examples