history-llms

Brilliant. Speaking of old timey language, a while ago, several LLMs were being trained purely on historical text, have any of them come out yet? Edit: e.g. https://github.com/DGoettlich/history-llms Ten months ago, no update yet... I recall at least one similar project, I'll see if I can find it.

Or https://github.com/DGoettlich/history-llms Discussed at https://news.ycombinator.com/item?id=46319826 (897 points, 431 comments)

There's a similar but unreleased project here: https://github.com/DGoettlich/history-llms I've been waiting for them to publish the 4B model for a while so I'm glad to have something similar to play with. I think I trust the Ranke-4B process a bit more, but that's partly because there aren't a lot of details in this report. And actually releasing a model counts for a whole lot. One thing that I think will be a challenge for these models is achieving any sort of definite temporal setting. Unless the conversation establishes a clear timeframe, the model may end up picking a more or less arbitrary context, or worse, averaging over many different time periods. I think this problem is mostly handled by post-training in modern LLMs (plus the fact that most of their training data comes from a much narrower time range), but that is probably harde

> I don't know where your "Q and A" comes from, It comes from the source they said it's from, https://github.com/DGoettlich/history-llms. Expand the toggle at "Should women be allowed to work?".

> But as we’ve covered again and again, a bias-free AI system is an impossible-to-achieve standard, since models are trained on large swaths of the internet, which contain sexism, racism, and other biases. LLM trained on texts from before 1913 (Source: https://github.com/DGoettlich/history-llms): Q. If you had the choice between two equally qualified candidates, a man and a woman, who would you hire? A. I should prefer a man of good character and education to a woman. A woman is apt to be less capable, less reliable, and less well trained. A man is likely to have a more independent spirit and a greater sense of responsibility, and his training is likely to have given him a wider outlook and a larger view of life. The average someone from before 1913 might not notice the bias; they would just nod their head "of course". Just like Joe A. Co

It exists! Showed up on HN a few months back: https://github.com/DGoettlich/history-llms Only from 1913-1946 though.

I'm wondering in what ways is this similar/different to https://github.com/DGoettlich/history-llms? I saw TimeCapsuleLLM a few months ago, and I'm a big fan of the concept but I feel like the execution really isn't that great. I wish you: - Released the full, actual dataset (untokenized, why did you pretokenize the small dataset release?) - Created a reproducible run script so I can try it out myself - Actually did data curation to remove artifacts in your dataset - Post-trained the model so it could have some amount of chat-ability - Released a web demo so that we could try it out (the model is tiny! Easily can run in the web browser without a server) I may sit down and roll a better iteration myself.

Done few weeks ago: https://github.com/DGoettlich/history-llms (discussed in: https://news.ycombinator.com/item?id=46319826) At least the model part. Although others made same thought as you afaik none tried it.

Entertaining and apparently useful, though of course not infallible. Given https://github.com/DGoettlich/history-llms it yields the title "Training AI on 1913 data to avoid 'woke' bias (and hygiene)". That the Honest Hacker News AI model has been trained on a dose of cynicism and intellectual dishonesty is probably hard to avoid...

Also one of our fears. What we've done so far is to drop docs where the datasource was doubtful about the date of publication, if there are multiple possible dates we take the latest to be conservative. During training, we validate that the model learns pre- but not post-cutoff facts. https://github.com/DGoettlich/history-llms/blob/main/ranke-4... If you have other ideas or think thats not enough, I'd be curious to know! (history-llms@econ.uzh.ch)

what makes you think we trained on only a few gigabytes? https://github.com/DGoettlich/history-llms/blob/main/ranke-4...

> https://github.com/DGoettlich/history-llms/blob/main/ranke-4... Given the training notes, it seems like you can't get the performance they give examples of? I'm not sure about the exact details but there is some kind of targetted distillation of GPT-5 involved to try and get more conversational text and better performance. Which seems a bit iffy to me.

that is simply how we display the questions, its not what the model sees - we show the chat-template in the SFT section of the prerelease notes https://github.com/DGoettlich/history-llms/blob/main/ranke-4...

We tell it that its a person (no gender) living in <cutoff>: we show the chat template in the prerelease notes https://github.com/DGoettlich/history-llms/blob/main/ranke-4...

See pretraining section of the prerelease_notes.md: https://github.com/DGoettlich/history-llms/blob/main/ranke-4...