ATatrudeau·Aug 27, 2018Submission
Spearmint – Open-source black-box optimization
2 points · Open discussionJasperSnoek
spearmint
Python2013–2018
JasperSnoek / spearmint
spearmint
Spearmint is a package to perform Bayesian optimization according to the algorithms outlined in the paper: Practical Bayesian Optimization of Machine Learning Algorithms. Jasper Snoek, Hugo Larochelle and Ryan P. Adams. Advances in Neural Information Processing Systems, 2012
11Distinct people
12Mentions
0Explicit recommenders
1.4KGitHub stars
What to know before exploring
What it is
Spearmint is a package to perform Bayesian optimization according to the algorithms outlined in the paper: Practical Bayesian Optimization of Machine Learning Algorithms. Jasper Snoek, Hugo Larochelle and Ryan P. Adams. Advances in Neural Information Processing Systems, 2012
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year12 dated mentions
Observed mentions: 2013-09-18 14:20:38 UTC to 2018-08-27 21:03:00 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2013 | 2013-01-01 00:00:00 UTC | 2014-01-01 00:00:00 UTC | 2 | 1 | 1 |
| 2014 | 2014-01-01 00:00:00 UTC | 2015-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 3 | 1 | 2 |
What Hacker News says
12 sourcesOne of their more interesting applications is in Bayesian Optimization (BO). You'd do BO optimization where evaluating the objective function is expensive. As against, lets say, where you might use gradient descent, where you navigate your space via repeated evaluation of the objective function.
BO essentially starts building its own model of what the objective function looks like in relation to its parameters, and picks a few promising points to evaluate it at. Thus, it moves the burden of evaluating a costly objective multiple times to identifying these highly promising points.
A BO algorithm can use GP to build its own model since the uncertainty estimates that come with it are fairly valuable and help in this seeking of promising points. For ex the GP can say that at so-and-so value of the parameter the objective function is expected
Read the rest
to do well, and its confident about it, hence this point can be explored next ... or it can say that at a particular region its extremely uncertain of its estimates, hence the region needs exploring. The BO keeps updating its model everytime the objective function is evaluated. Since at any instant the model encodes knowledge from all past evaluations, this is also sometimes known as Sequential Model Based Optimization (SMBO).
If you need a practical example of its use within Machine Learning, BO with GPs have been used in finding the right hyperparameters for a model - [1], [2]. Traditionally you'd perform a grid-search in the space of hyperparameters, building a model at all points on the grid; BO tells you that you don't need to really do that --- you can be smart of what points in the grid should you actually build your model at.
The problem with GP based BO is scaling. There has been a fair amount of research (its an active area) to address this - sometimes by performing BOs without GPs, and sometimes using GPs but trying to speed it up [3].
[1] A good tutorial on the approach - https://arxiv.org/abs/1012.2599
[2] A library for this https://github.com/JasperSnoek/spearmint The corresponding paper is linked.
[3] This paper is a good example - it suggests doing BO both using GPs (in this case suggesting speed-ups, Section 3) and without it (section 4) https://papers.nips.cc/paper/4443-algorithms-for-hyper-param... . In fact [2] is also a good example of speeding up GP-based BO.
You are not doing away with the test/validation set. You are doing away with:
1. grid search - you would now only test for parameters that look promising (based on some criteria). Instead of covering the whole grid, you want to be smart about which points to try out. [1]
2. you might still do grid search but you want to preferentially allocate resources to parameter exploration based on how promising they seem. Hyperband - [2]- is an example.
3. you can do both - you can be smart about picking parameters, but for the ones you pick you can preferentially allocate resources. Ex *Bayesian Optimization and Hyperband* (BOHB)
[1] Bayesian Optimization using Gaussian Processes is an example of this. https://arxiv.org/pdf/1206.2944.pdf Here's a library that helps you do this: https://github.com/JasperSnoek/spearmint. But there are othe
Read the rest
r techniques in this family like Randomized Online Aggressive Racing (ROAR), Deep Network for Global Optimization (DNGO), Tree-Structured Parzen Estimators (TPE), etc
[2] Hyperband: https://arxiv.org/abs/1603.06560
[3] BOHB: https://arxiv.org/abs/1807.01774
As you pointed out, it is all about a balance, and every feature has different tradeoffs.
SigOpt was designed to unlock the power of Bayesian optimization for anyone doing machine learning. We believe that you shouldn't need to be an expert and spend countless hours of administration to achieve great results for every model. We're wrapping an ensemble of the best Bayesian methods behind a simple interface [0] and constantly making improvements so that people can focus on designing features and their individual domain expertise, instead of needing to build and maintain their own hyperparameter optimization tools to see the benefit.
For experts who want to spend a lot of time and effort customizing, administering, updating, and maintaining a hyperparameter tuning solution I would recommend forking one of the open source packages out there l
Read the rest
ike spearmint [1] or MOE [2] (disclaimer, I wrote MOE while working at Yelp).
[0]: https://sigopt.com/docs
[1]: https://github.com/JasperSnoek/spearmint
[2]: https://github.com/Yelp/MOE
In short, these guys are using deep neural nets to find good hyperparameters for training other deep neural nets, and this works as well as a Gaussian Process[1] but is more scalable and can be parallelized, allowing for faster optimization of hyperparameters.
--
[1] For example, like Spearmint: https://github.com/JasperSnoek/spearmint
In Scalable Bayesian Optimization Using Deep Neural NetworksI just came across this today. Code for the project is here:
https://github.com/JasperSnoek/spearmint
Based on the paper (preprint): http://arxiv.org/pdf/1206.2944.pdf
In Whetlab.com: API for Baysian optimization of hyperparametersSpearmint, for example: https://github.com/JasperSnoek/spearmint/blob/master/spearmi...
In How to Evaluate Machine Learning Models: Hyperparameter TuningJust to add a couple others:
rmsprop is a great technique I don't hear talked about as much, example implementation here: https://github.com/BRML/climin/blob/master/climin/rmsprop.py
Using nesterov momentum and a "sparse" weight initialization scheme rather than uniform: https://www.cs.toronto.edu/~hinton/absps/momentum.pdf
Reducing the learning rate exponentially and increasing the momentum rate linearly over the course of training. Learning rate from .5 to .0001, momentum from .7 to .995. I've seen variations on this, like adjusting based on sigmoid curve.
Dropout may or may not help, adjusting dropout rate (percentage of activations that are discarded) may or may not help.
Mini-batch size can make a difference. Somewhere between 2 and 200?
You can use bayesian optimization to intelligently search hyperparameters: https://github.com/Jas
Read the rest
perSnoek/spearmint
Try rmsprop though, I've heard good things.
Neat!
For those interested in this sort of thing, a couple of other libraries with a similar range of uses:
https://github.com/hyperopt/hyperopt
https://github.com/JasperSnoek/spearmint
Would be interested if anyone has any thoughts on how these compare (I've been quite impressed so far by hyperopt but haven't tried the others yet).
In Show HN: Moe, a Metric Optimization EngineFor those that don't know: a Bayesian optimization approach based on Gaussian processes has been implemented in Python by the author of that paper [0].
Here's a nice blog post with some information on some problems it can run into in practice [1].
[0]: https://github.com/JasperSnoek/spearmint
[1]: http://fastml.com/tuning-hyperparams-automatically-with-spea...
In Hyperopt: Optimize your machine learning parameters automagically