LLMLingua

This is my way to do it: 1. Archive whole repository into single text file with Repopack: https://github.com/yamadashy/repomix 2. To reduce token, compress the file with LLMLingua-2: https://github.com/microsoft/LLMLingua (fewer token = more context can be given to LLM = LLM better understands your repository) 3. Copy & Paste the compressed archive text contents as a context, into ChatGPT’s input field as-is, or local LLMs. 4. Ask the LLM for documentation generation. Like, “this is a repository source code. given context, generate a ‘table-of-content’.” Then you will get a ToC. If it looks good, you can ask for generating first chapter. And keeps going until you finish whole documentation. If you are trying to document Typescript/Javascript codebase, You may use bundlers like esbuild for step 2, which will beneficial for token reducing.

Hey HN, Like (perhaps?) many of you working with different open-weight LLMs, I often find myself wanting a quick way to check token counts for specific Hugging Face models without having to write a Python script each time. I often find myself googling something like "online Qwen tokenizer" and coming up empty-handed. So, I built Tokiwi (https://tokiwi.dev), the online tokenizer I've been looking for. What it does: - You paste your text. - You enter any model repository ID from Hugging Face (like `google/gemma-3-27b-it`, `deepseek-ai/DeepSeek-V3-0324`, etc.). - It fetches the tokenizer from HF and shows you the token count and the list of tokens. - It also supports gated models if you provide your HF token (handled securely via the backend, not stored). It's pretty straightforward right now. Built with Next.js on Vercel for the frontend an