There's a saying: lies, damn lies and benchmarks. In this case AFAIK mi one has replicated these claims.
MS hasn't got it in the standard CNN inference benchmark yet though: https://github.com/soumith/convnet-benchmarks. This is the benchmark that showed how slow TensorFlow was on non-Google libraries until Google fixed it.
In Cognitive Toolkit beta for deep learning advancessoumith
convnet-benchmarks
Python2014–2025
soumith / convnet-benchmarks
convnet-benchmarks
Easy benchmarking of all publicly accessible implementations of convnets
29Distinct people
36Mentions
0Explicit recommenders
2.7KGitHub stars
What to know before exploring
What it is
Easy benchmarking of all publicly accessible implementations of convnets
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- autumnai/deep-learning-benchmarks · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 12:01:02 AM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year36 dated mentions
Observed mentions: 2014-09-29 18:59:16 UTC to 2025-11-07 05:52:19 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2014 | 2014-01-01 00:00:00 UTC | 2015-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 9 | 2 | 7 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 20 | 2 | 18 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 1 | 1 | 0 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
convnet-benchmarks
Easy benchmarking of all public open-source implementations of convnets.
A summary is provided in the section below.
Machine: 6-core Intel Core i7-5930K CPU @ 3.50GHz + NVIDIA Titan X + Ubuntu 14.04 x86_64
Imagenet Winners Benchmarking
I pick some popular imagenet models, and I clock the time for a full forward + backward pass. I average my times over 10 runs. I ignored dropout and softmax layers.
Notation
Input is described as {batch_size}x{num_filters}x{filter_width}x{filter_height}. Where batch_size is the number of images used in a minibatch, num_filters is the number of channels in an image, filter_width is the width of the image, and filter_height is the height of the image.
One small note:
The CuDNN benchmarks are done using Torch bindings. One can also do the same via Caffe bindings or bindings of any other library. This note is here to clarify that Caffe (native) and Torch (native) are the convolution kernels which are present as a default fallback. Some of the frameworks like TensorFlow and Chainer are benchmarked with CuDNN, but it is not explicitly mentioned, and hence one might think that these frameworks as a whole are faster, than for example Caffe, which might not be the case.
AlexNet (One Weird Trick paper) - Input 128x3x224x224
Library
Class
Time (ms)
forward (ms)
backward (ms)
CuDNN[R4]-fp16 (Torch)
cudnn.SpatialConvolution
71
25
46
Nervana-neon-fp16
ConvLayer
78
25
52
CuDNN[R4]-fp32 (Torch)
cudnn.SpatialConvolution
81
27
53
TensorFlow
conv2d
81
26
55
Nervana-neon-fp32
ConvLayer
87
28
58
fbfft (Torch)
fbnn.SpatialConvolution
104
31
72
Chainer
Convolution2D
177
40
136
cudaconvnet2*
ConvLayer
177
42
135
CuDNN[R2] *
cudnn.SpatialConvolution
231
70
161
Caffe (native)
ConvolutionLayer
324
121
203
Torch-7 (native)
SpatialConvolutionMM
342
132
210
CL-nn (Torch)
SpatialConvolutionMM
963
388
574
Caffe-CLGreenTea
ConvolutionLayer
1442
210
1232
Overfeat [fast] - Input 128x3x231x231
Library
Class
Time (ms)
forward (ms)
backward (ms)
Nervana-neon-fp16
ConvLayer
176
58
118
Nervana-neon-fp32
ConvLayer
211
69
141
CuDNN[R4]-fp16 (Torch)
cudnn.SpatialConvolution
242
86
156
CuDNN[R4]-fp32 (Torch)
cudnn.SpatialConvolution
268
94
174
TensorFlow
conv2d
279
90
189
fbfft (Torch)
SpatialConvolutionCuFFT
342
114
227
Chainer
Convolution2D
620
135
484
cudaconvnet2*
ConvLayer
723
176
547
CuDNN[R2] *
cudnn.SpatialConvolution
810
234
576
Caffe
ConvolutionLayer
823
355
468
Torch-7 (native)
SpatialConvolutionMM
878
379
499
CL-nn (Torch)
SpatialConvolutionMM
963
388
574
Caffe-CLGreenTea
ConvolutionLayer
2857
616
2240
OxfordNet [Model-A] - Input 64x3x224x224
Library
Class
Time (ms)
forward (ms)
backward (ms)
Nervana-neon-fp16
ConvLayer
254
82
171
Nervana-neon-fp32
ConvLayer
320
103
217
CuDNN[R4]-fp16 (Torch)
cudnn.SpatialConvolution
471
140
331
CuDNN[R4]-fp32 (Torch)
cudnn.SpatialConvolution
529
162
366
TensorFlow
conv2d
540
158
382
Chainer
Convolution2D
885
251
632
fbfft (Torch)
SpatialConvolutionCuFFT
1092
355
737
cudaconvnet2*
ConvLayer
1229
408
821
CuDNN[R2] *
cudnn.SpatialConvolution
1099
342
757
Caffe
ConvolutionLayer
1068
323
745
Torch-7 (native)
SpatialConvolutionMM
1105
350
755
CL-nn (Torch)
SpatialConvolutionMM
3437
875
2562
Caffe-CLGreenTea
ConvolutionLayer
5620
988
4632
GoogleNet V1 - Input 128x3x224x224
Library
Class
Time (ms)
forward (ms)
backward (ms)
Nervana-neon-fp16
ConvLayer
230
72
157
Nervana-neon-fp32
ConvLayer
270
84
186
TensorFlow
conv2d
445
135
310
CuDNN[R4]-fp16 (Torch)
cudnn.SpatialConvolution
462
112
349
CuDNN[R4]-fp32 (Torch)
cudnn.SpatialConvolution
470
130
340
Chainer
Convolution2D
687
189
497
Caffe
ConvolutionLayer
1935
786
1148
CL-nn (Torch)
SpatialConvolutionMM
7016
3027
3988
Caffe-CLGreenTea
ConvolutionLayer
9462
746
8716
Layer-wise Benchmarking (Last Updated April 2015)
Spatial Convolution layer (3D input 3D output, densely connected)
forward + backprop (wrt input and weights)
Original Library
Class/Function Benchmarked
Time (ms)
forward (ms)
backward (ms)
fbfft
SpatialConvolutionCuFFT
256
101
155
cuda-convnet2 *
ConvLayer
977
201
776
cuda-convnet**
pylearn2.cuda_convnet
1077
312
765
CuDNN R2 *
cudnn.SpatialConvolution
1019
269
750
Theano
CorrMM
1225
407
818
Caffe
ConvolutionLayer
1231
396
835
Torch-7
SpatialConvolutionMM
1265
418
877
DeepCL
ConvolutionLayer
6280
2648
3632
cherry-picking****
best per layer
235
79
155
This table is NOT UPDATED For TITAN-X. These numbers below were on Titan Black and are here only for informational and legacy purposes.
Original Library
Class/Function Benchmarked
Time (ms)
forward (ms)
backward (ms)
Theano (experimental)***
conv2d_fft
1178
304
874
Torch-7
nn.SpatialConvolutionBHWD
1892
581
1311
ccv
ccv_convnet_layer
809+bw
809
Theano (legacy)
conv2d
70774
3833
66941
* indicates that the library was tested with Torch bindings of the specific kernels.
** indicates that the library was tested with Pylearn2 bindings.
*** This is an experimental module which used FFT to calculate convolutions. It uses a lot of memory according to @benanne
**** The last row shows results obtainable when choosing the best-performing library for each layer.
L1 - Input: 128x128 Batch-size 128, Feature maps: 3->96, Kernel Size: 11x11, Stride: 1x1
L2 - Input: 64x64 Batch-size 128, Feature maps: 64->128, Kernel Size: 9x9, Stride: 1x1
L3 - Input: 32x32 Batch-size 128, Feature maps: 128->128, Kernel Size: 9x9, Stride: 1x1
L4 - Input: 16x16 Batch-size 128, Feature maps: 128->128, Kernel Size: 7x7, Stride: 1x1
L5 - Input: 13x13 Batch-size 128, Feature maps: 384->384, Kernel Size: 3x3, Stride: 1x1
The table is ranked according to the total time forward+backward calls for layers (L1 + L2 + L3 + L4 + L5)
Breakdown
forward
Columns L1, L2, L3, L4, L5, Total are times in milliseconds
Original Library
Class/Function Benchmarked
L1
L2
L3
L4
L5
T
Read the full README on GitHubWhat Hacker News says
20 sourcesThere is another nice one here:
https://github.com/soumith/convnet-benchmarks
In Simple Deep Learning BenchmarkWhy can't Torch utilize more threads in CPU cores?
Taken from the article itself:
"both of them cannot run normally when threads usage is set to be bigger than the number of CPU cores on desktop CPU."
Do the authors set up the system correctly?
You're right that Torch is faster than TensorFlow in RNN. But Torch is slower than TesnorFlow in AlexNet and ResNet.
There is a set of benchmarks for many DL approaches as found in https://github.com/soumith/convnet-benchmarks
In Benchmarking State-Of-the-Art Deep Learning Software ToolsNot even wrong. I have two PCs with 4 Titan X (Maxwell) GPUs and a third PC with 4 Titan X (Pascal) GPUs. Both of these systems are available today (I built them myself, total BOM about $7K), and both will destroy 4 Xeon Phi servers at Deep Learning.
The benchmark Intel presented here is as disingenuous as their infamous white paper from 2010: http://pcl.intel-research.net/publications/isca319-lee.pdf
In comparison, a single Knights Landing Xeon Phi will be ~$7K. I know where I put my money. Caveat Emptor.
But Xeon Phi and I go way back here. They've been trying to beat my AMBER GPU code since 2013 or so. Many man years later I believe that a Knight's Corner is now ~35% faster than 2 Xeon CPUs with 1M atoms or more (source: http://adsabs.harvard.edu/abs/2016CoPhC.201...95N)
Meanwhile, the CUDA code has continued to scale with the GP
Read the rest
U roadmap and a Titan XP is arguably 9-10x faster than 2 Xeon CPUs. No data is supplied at the low-end for Xeon Phi and I think we can safely assume it's because performance there sucks. (source: http://ambermd.org/gpus/benchmarks.htm)
Xeon Phi? IMO avoid avoid avoid until they start winning head to head 3rd party benchmarking fights like Soumith Chintala's fantastic convnet benchmark data: https://github.com/soumith/convnet-benchmarks
No, it's not. Why does TensorFlow have separate benchmarks for Soumith's convnet comparison that show it to be significantly slower?
https://github.com/soumith/convnet-benchmarks
In Google Sprints Ahead in AI Building Blocks, Leaving Rivals WaryWhat's Deep Learning 4J's performance on the Soumith benchmarks?
https://github.com/soumith/convnet-benchmarks
In Generating Recommendations at Amazon Scale with Apache Spark and Amazon DSSTNEThese are the benchmarks supported by Soumith Chintala (Facebook AI Research):
https://github.com/soumith/convnet-benchmarks
They place TensorFlow performance on par with Torch (within 10%).
In TensorFlow – Consise Examples for BeginnersAll of the comparable tools are practically locked into nVidia gpus/CUDA. TensorFlow is rapidly reaching performance parity on that hardware [1] and can now be run on this so it's actually sort of the least locked in framework.
[1] It has been climbing the charts at https://github.com/soumith/convnet-benchmarks for example.
In Google supercharges machine learning tasks with TPU custom chipArria 10: http://www.nextplatform.com/2015/08/27/microsoft-extends-fpg...
600 images/s inference (highest ever publically reported), projected to be as high as 900 images/s. ~$5000 per FPGA.
Compare and contrast with TitanX/GTX 980 TI now topping 5000 images/s. GTX 1080 will only be faster than this and $600.
https://github.com/soumith/convnet-benchmarks
And so far, no FPGA training numbers. But here are some distributed training numbers for $10Kish Xeon servers:
https://software.intel.com/en-us/articles/caffe-training-on-...
64 of them train Alexnet in ~5 hours. A DGX-1 does it in 2. Don't have/Can't get a $129K DGX-1? Fine, buy a Big Sur from one of several vendors and throw in 8 GTX 1080s as soon as they're out, implement "One Weird Trick", and you'll do it in <3. That ought to run you about $30-$35K versus >$600K for those 64 Xe
Read the rest
on servers.
FPGAs OTOH are fantastic at low memory bandwidth embarrassingly parallel computation like bitcoin mining. Deep Learning is not such a domain.
The most interesting thing about AMD is their ability to create GPUs with superior synthetic specs than their NVIDIA competitor and yet they lose across the board in anything but embarrassingly parallel applications. Not only do they lose, but they lose big.
Here are two examples. First, the engine behind Folding@Home:
https://simtk.org/plugins/moinmoin/openmm/BenchmarkOpenMMDHF...
See also AlexNet (Deep Learning's most prominent benchmark):
https://github.com/amd/OpenCL-caffe
versus GTX TitanX:
https://github.com/soumith/convnet-benchmarks
TLDR: AMD losing by a factor of 2 or more...
So unless this changes dramatically, NVIDIA IMO will continue to dominate.
But hey, they both crush Xeon Phi and FPGAs so it doesn't suck to be #2 when #3 is Intel.
In NVIDIA Announces the GeForce GTX 1000 SeriesAnd even those numbers on the front page are out of date :) (we're even faster now: https://github.com/soumith/convnet-benchmarks/pull/96, which is from a few weeks ago.)
The field is moving quickly enough that many published benchmarks are stale within 3 months, and it's a lot of hard work to maintain up to date benchmarks, given how many frameworks there are. Also there are performance/memory/scalability/flexibility tradeoffs everywhere, so it's hard to capture everything in one number without a tremendous number of caveats.
In Announcing TensorFlow 0.8 – now with distributed computing supportThat study's way out of date - it benchmarked the CuDNNv2 version. Soumith's convnet-benchmarks is much more up-to-date: https://github.com/soumith/convnet-benchmarks
but it hasn't yet been updated to reflect the latest performance improvements in 0.8. We've continued to push on both single-machine and distributed performance, and the next update to soumith's benchmarks should continue to show that improvement.
In Announcing TensorFlow 0.8 – now with distributed computing supportThe Nervana 16bit floating point implementations is probably worth noting in this context.
They perform (speed-wise) pretty well: https://github.com/soumith/convnet-benchmarks
In XNOR-Net: ImageNet Classification Using Binary Convolutional Neural NetworksYes, on their site they post Torch CuDNN v4 as faster than Leaf [0]. Seems exciting for an early release.
Can it get much faster than something like Torch? I would think if CuDNN is doing most of the computation time it would be hard to see big improvements. Perhaps go the route of Neon and tune your GPGPU code like crazy [1, 2], or MXNet and think about distributed computing performance [3].
[0] http://autumnai.com/deep-learning-benchmarks
[1] https://github.com/soumith/convnet-benchmarks
[2] https://github.com/NervanaSystems/neon
[3] http://alex.smola.org/talks/NIPS15.pdf
In Leaf: Machine learning framework in RustYou are right, batchsize is important and we should make that more clear.
The numbers in the benchmark are taken from our deep-learning-benchmarks[1] which we are still in the process of building up. It might actually make sense to test the same model with different batch sizes. The current benchmarks are based on the convnet-benchmarks[2] where the Alexnet model has a batch size of 128. (Alexnet was chosen because out of the benchmarks that's the one I am most familiar with, since it small enough that I can work with it on my Laptop)
In some informal tests Leaf was generally faster than other frameworks in smaller batch sizes, but no benchmarks that we could publish with confidence yet.
[1]: https://github.com/autumnai/deep-learning-benchmarks
[2]: https://github.com/soumith/convnet-benchmarks
In Leaf: Machine learning framework in RustShowing the 15 most recent of 20 sources. Enable JavaScript to search and browse every indexed mention.