HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
kingoflolz
mesh-transformer-jax
Python2021–2023
kingoflolz / mesh-transformer-jax

mesh-transformer-jax

Model parallel transformers in JAX and Haiku

14Distinct people
15Mentions
0Explicit recommenders
6.4KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Model parallel transformers in JAX and Haiku

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 6:34:31 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month15 dated mentions

Observed mentions: 2021-06-09 04:16:48 UTC to 2023-03-22 15:41:33 UTC.

012342021-06-01 00:00:00 UTC to 2021-07-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsJun 20212021-07-01 00:00:00 UTC to 2021-08-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments2021-08-01 00:00:00 UTC to 2021-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-09-01 00:00:00 UTC to 2021-10-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2021-10-01 00:00:00 UTC to 2021-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-11-01 00:00:00 UTC to 2021-12-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2021-12-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-01-01 00:00:00 UTC to 2022-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-02-01 00:00:00 UTC to 2022-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-03-01 00:00:00 UTC to 2022-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-04-01 00:00:00 UTC to 2022-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsApr 20222022-05-01 00:00:00 UTC to 2022-06-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2022-06-01 00:00:00 UTC to 2022-07-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2022-07-01 00:00:00 UTC to 2022-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-08-01 00:00:00 UTC to 2022-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-09-01 00:00:00 UTC to 2022-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-10-01 00:00:00 UTC to 2022-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-11-01 00:00:00 UTC to 2022-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-12-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments2023-01-01 00:00:00 UTC to 2023-02-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-02-01 00:00:00 UTC to 2023-03-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-03-01 00:00:00 UTC to 2023-04-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsMar 2023

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jun 20212021-06-01 00:00:00 UTC2021-07-01 00:00:00 UTC101
Jul 20212021-07-01 00:00:00 UTC2021-08-01 00:00:00 UTC312
Aug 20212021-08-01 00:00:00 UTC2021-09-01 00:00:00 UTC000
Sep 20212021-09-01 00:00:00 UTC2021-10-01 00:00:00 UTC110
Oct 20212021-10-01 00:00:00 UTC2021-11-01 00:00:00 UTC000
Nov 20212021-11-01 00:00:00 UTC2021-12-01 00:00:00 UTC101
Dec 20212021-12-01 00:00:00 UTC2022-01-01 00:00:00 UTC000
Jan 20222022-01-01 00:00:00 UTC2022-02-01 00:00:00 UTC000
Feb 20222022-02-01 00:00:00 UTC2022-03-01 00:00:00 UTC000
Mar 20222022-03-01 00:00:00 UTC2022-04-01 00:00:00 UTC000
Apr 20222022-04-01 00:00:00 UTC2022-05-01 00:00:00 UTC000
May 20222022-05-01 00:00:00 UTC2022-06-01 00:00:00 UTC101
Jun 20222022-06-01 00:00:00 UTC2022-07-01 00:00:00 UTC101
Jul 20222022-07-01 00:00:00 UTC2022-08-01 00:00:00 UTC000
Aug 20222022-08-01 00:00:00 UTC2022-09-01 00:00:00 UTC000
Sep 20222022-09-01 00:00:00 UTC2022-10-01 00:00:00 UTC000
Oct 20222022-10-01 00:00:00 UTC2022-11-01 00:00:00 UTC000
Nov 20222022-11-01 00:00:00 UTC2022-12-01 00:00:00 UTC000
Dec 20222022-12-01 00:00:00 UTC2023-01-01 00:00:00 UTC404
Jan 20232023-01-01 00:00:00 UTC2023-02-01 00:00:00 UTC101
Feb 20232023-02-01 00:00:00 UTC2023-03-01 00:00:00 UTC101
Mar 20232023-03-01 00:00:00 UTC2023-04-01 00:00:00 UTC101

What Hacker News says

1 sources

Fine-tuning GPT-J: key takeaways

Hello all, We've spent quite some time benchmarking the best fine-tuning techniques for GPT-J at NLP Cloud (https://nlpcloud.io). Finding the best solution was not straightforward and we had to look at things like speed, server costs, ease of development, accuracy of the fine-tuned model... It took time but we ended up with a nice setup (and we are now officially proposing GPT-J fine-tuning + automatic deployment on our platform). Here are our key takeaways: - The best methodology seems to be the one from the Mesh Transformer Jax team: https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md - Fine-tuning on GPU is not ideal. Even several GPUs used in parallel with Deepspeed can be very slow. We used 4 GPUs Tesla T4 in parallel, and it took 1h30 to only compute our first checkpoint (+ 80GB of RAM used...), for a tr
Read the rest
aining dataset made up of 20k examples. Maybe a GPU A100 would be worth a try. - Fine-tuning on TPU is very efficient but it takes a TPU v3 because TPUs v2 are running out of memory. It takes around 15mns, for a training dataset made up of 20k examples, which is really awesome. - The overall process is not straightforward as it takes several kind of conversions (converting the datasets to the right format, making a slim version of the model, converting the weights to Transformers...) In the end this is worth the effort, because combining fine-tuning and few-shot learning makes GPT-J very impressive and suited for all sorts of use cases. If you guys have different feedbacks about GPT-J fine-tuning, please don't hesitate to comment, I would love to have your opinion. Hope you found the above useful!
1 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.