HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
OpenNMT
CTranslate2
C++2023–2025
OpenNMT / CTranslate2

CTranslate2

Fast inference engine for Transformer models

AI & Machine LearningSystems & InfrastructureLearning & ReferenceNetworking
Explore on GitHubRead Hacker News discussions
7Distinct people
12Mentions
0Explicit recommenders
4.7KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Fast inference engine for Transformer models

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month12 dated mentions

Observed mentions: 2023-02-20 18:50:39 UTC to 2025-01-17 23:06:21 UTC.

02462023-02-01 00:00:00 UTC to 2023-03-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsFeb 20232023-03-01 00:00:00 UTC to 2023-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-01 00:00:00 UTC to 2023-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-05-01 00:00:00 UTC to 2023-06-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-06-01 00:00:00 UTC to 2023-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-07-01 00:00:00 UTC to 2023-08-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-08-01 00:00:00 UTC to 2023-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 6 mentions; 0 submissions, 6 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsJan 20242024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsJan 2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Feb 20232023-02-01 00:00:00 UTC2023-03-01 00:00:00 UTC101
Mar 20232023-03-01 00:00:00 UTC2023-04-01 00:00:00 UTC000
Apr 20232023-04-01 00:00:00 UTC2023-05-01 00:00:00 UTC000
May 20232023-05-01 00:00:00 UTC2023-06-01 00:00:00 UTC110
Jun 20232023-06-01 00:00:00 UTC2023-07-01 00:00:00 UTC000
Jul 20232023-07-01 00:00:00 UTC2023-08-01 00:00:00 UTC101
Aug 20232023-08-01 00:00:00 UTC2023-09-01 00:00:00 UTC000
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC000
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC606
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC202
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC000
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC000
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC000
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC000
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC000
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC000
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC000
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC000
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC101

What Hacker News says

12 sources
kkielhofner·Oct 9, 2023
We'd love to move beyond Nvidia. The issue (among others) is we achieve the speech recognition performance we do largely thanks to ctranslate2[0]. They've gone on the record saying that they essentially have no interest in ROCm[1]. Of course with open source anything is possible but we see this as being one of several fundamental issues in supporting AMD GPGPU hardware. [0] - https://github.com/OpenNMT/CTranslate2 [1] - https://github.com/OpenNMT/CTranslate2/issues/1072
In A Raspberry Pi 5 is better than two Pi 4S
kkielhofner·Oct 7, 2023
> While I agree that it's much more effort to get things working on AMD cards than it is with Nvidia, I was a bit surprised to see this comment mention Whisper being an example of "5-10x as performant". It easily is. See the benchmarks[0] from faster-whisper which uses Ctranslate2. That's 5x faster than OpenAI reference code on a Tesla V100. Needless to say something like a 4080 easily multiplies that. > https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080, albeit with higher power draw. To be fair it's not using ROCm but Direct3D 11, but for performance/price arguments sake that detail is not relevant. With all due respect to the author of the article this is "my first entry
Read the rest
into ML" territory. They talk about a 5-10 second delay, my project can do sub 1 second times[1] even with ancient GPUs thanks to Ctranslate2. I don't have an RTX 4080 but if you look at the performance stats for the closest thing (RTX 4090) the performance numbers are positively bonkers - completely untouchable for anything ROCm based. Same goes for the other projects I linked, lmdeploy does over 100 tokens/s in a single session with LLama2 13b on my RTX 4090 and almost 600 tokens/s across eight simultaneous sessions. > EDIT: Also using CTranslate2 as an example is not great as it's actually a good showcase why ROCm is so far behind CUDA: It's all about adapting the tech and getting the popular libraries to support it. Things usually get implemented in CUDA first and then would need additional effort to add ROCm support that projects with low amount of (possibly hobbyist) maintainers might not have available. There's even an issue in CTranslate2 where they clearly state no-one is working to get ROCm supported in the library. ( https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomm... ) I don't understand what you're saying here. It (along with the other projects I linked here[2]) are fantastic examples of just how far behind the ROCm ecosystem is. ROCm isn't even on the radar for most of them as your linked issue highlights. Things always get implemented in CUDA first (ten years in this space and I've never seen ROCm first) and ROCm users either wait months (minimum) for sub-par performance or never get it at all. [0] - https://github.com/guillaumekln/faster-whisper#benchmark [1] - https://heywillow.io/components/willow-inference-server/#ben... [2] - https://news.ycombinator.com/item?id=37793635#37798902
In AMD may get across the CUDA moat
errnoh·Oct 7, 2023
While I agree that it's much more effort to get things working on AMD cards than it is with Nvidia, I was a bit surprised to see this comment mention Whisper being an example of "5-10x as performant". https://www.tomshardware.com/news/whisper-audio-transcriptio... is a good example of Nvidia having no excuses being double the price when it comes to Whisper inference, with 7900XTX being directly comparable with 4080, albeit with higher power draw. To be fair it's not using ROCm but Direct3D 11, but for performance/price arguments sake that detail is not relevant. EDIT: Also using CTranslate2 as an example is not great as it's actually a good showcase why ROCm is so far behind CUDA: It's all about adapting the tech and getting the popular libraries to support it. Things usually get implemented in CUDA first and then would need additional ef
Read the rest
fort to add ROCm support that projects with low amount of (possibly hobbyist) maintainers might not have available. There's even an issue in CTranslate2 where they clearly state no-one is working to get ROCm supported in the library. ( https://github.com/OpenNMT/CTranslate2/issues/1072#issuecomm... )
In AMD may get across the CUDA moat
kkielhofner·Oct 7, 2023
I wouldn’t say ROCm code is “slower”, per se, but in practice that’s how it presents. References: https://github.com/InternLM/lmdeploy https://github.com/vllm-project/vllm https://github.com/OpenNMT/CTranslate2 You know what’s missing from all of these and many more like them? Support for ROCm. This is all before you get to the really wildly performant stuff like Triton Inference Server, FasterTransformer, TensorRT-LLM, etc. ROCm is at the “get it to work stage” (see top comment, blog posts everywhere celebrating minor successes, etc). CUDA is at the “wring every last penny of performance out of this thing” stage. In terms of hardware support, I think that one is obvious. The U in CUDA originally stood for unified. Look at the list of chips supported by Nvidia drivers and CUDA releases. Literally anything from at least the past 10 years t
Read the rest
hat has Nvidia printed on the box will just run CUDA code. One of my projects specifically targets Pascal up - when I thought even Pascal was a stretch. Cue my surprise when I got a report of someone casually firing it up on Maxwell when I was pretty certain there was no way it could work. A Maxwell laptop chip. It also runs just as well on an H100. THAT is hardware support.
In AMD may get across the CUDA moat
kkielhofner·Oct 7, 2023
Whisper is actually a great example of why Nvidia has such a stronghold on ML/AI and why it’s so difficult to compete. There’s getting something to “work”, which is often enough of a challenge with ROCm. Then there’s getting it to work well (next challenge). Then there’s getting it to work as well as Nvidia/CUDA. With Whisper, as one example, you should be running it with ctranslate2[0]. Of all the platforms on their supported list you won’t find ROCm. When you really start to look around you’ll find that ROCm is (at best) still very much in the “get it to work (sometimes)” stage. In most cases it’s still a long way away from getting it to work well, and even further away from making it actually competitive with Nvidia for serious use cases and applications. People get excited about the progress ROCm has made getting basic things to work
Read the rest
with PyTorch and this is good - progress is progress. But saving 20% on the hardware when the equivalent Nvidia product is often somewhere between 5-10x as performant (at a fraction of the development time) because of vastly superior software support you realize pretty quickly Nvidia is actually a bargain compared to AMD. I’m desperately rooting for Nvidia to have some actual competition but after six years of ROCm and my own repeated failed attempts to have it make any sense overall I’m only more and more skeptical that real competition in the space will come from AMD. [0] - https://github.com/OpenNMT/CTranslate2
In AMD may get across the CUDA moat
Palmik·Oct 4, 2023
I see a lot of people misunderstanding what this is about. What this allows is incrementally updating the attention cache. It does not allow the model to see beyond its usual attention window. As they explain in a README, only the tokens that fit into the usual window are considered -- so if you ask a question about a long book, it will only consider the last pages. But it can still be useful. Imagine this use case, where you have a chat conversation between Assistant and User. Assume that the inputs to get the next assistant response are just the past conversation turns (cut off to fit context window). So for turn 1 the input is: User: (user turn 1) For turn 2 the input is: User: (user turn 1) Assistant: (assistant turn 1) User: (user turn 2) Etc. Now, what this allows you to do is reuse the attention computed from the prev
Read the rest
ious turns (since the prefix is the same). In practice, people often have a system prompt before the conversation history, which (as far a I can tell) makes this technique not applicable (the input prefix will change as soon as the conversation history is long enough that we need to start dropping the oldest turns, otherwise the system prompt would get ignored). In such case, what you could do is to cache at least the system prompt. This is also possible with https://github.com/OpenNMT/CTranslate2/blob/2203ad5c8baf878a...
In StreamingLLM: Efficient streaming technique enable infinite sequence lengths
albertzeyer·Feb 20, 2023
Python is just the gluing language. All the heavy lifting happens in CUDA or CuBLAS or CuDNN or so. Most optimizations for saving memory is by using lower precision numbers (float16 or less), quantization (int8 or int4), sparsification, etc. But this is all handled by the underlying framework like PyTorch. There are C++ implementations but they optimize on different aspects. For example: https://github.com/OpenNMT/CTranslate2/
In Running large language models like ChatGPT on a single GPU

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.