THtheluk·Jun 6, 2024Submission
Show HN: Open-Source LLM Patch Streaming – Speed and Output Token Improvement
Every time I ask an LLM like ChatGPT to make some modifications to my large chunk of text, it starts streaming the full thing back to me.
Here are the most annoying things about this:
1. It consumes a ton of unnecessary output tokens
2. It is limited to 4k output tokens
3. Sometimes modifies parts that I did not tell it to
4. IT IS SLOOOW (imagine we wanted to change just a word at the beginning)
What if, we could just patch our changes using a diff? I mean this is nothing groundbreaking, but still it seems, as if no one actually built something reusable or open.
So here it is: LLM Patcher (kinda dumb name I know)
Here is how it works
1. The user provides a text and a find-and-replace query.
2. The text is split into lines and sentences.
3. Each line and sentence is then prefixed with an identifier that looks like <l1s1> for line 1, sente
Read the rest
nce 1.
4. The LLM is then asked to find-and-replace the query in each line and sentence.
5. The changes are then streamed back to the user in the form of a diff. The diff looks like <r:l1s1> string to find || string to replace.
Let me know what you think about it and please give it a star
Cheers, Luke