wdoc

If you have a TypeError it's already too late. Decorating everything with beartype means I can catch things way in advance. It also forces me to specify types to all my functions which really helps LLM. See for example in wdoc, my advanced personal RAG library: https://github.com/thiswillbeyourgithub/wdoc/blob/main/wdoc/...

No, pyright is called on static code. Beartime is called during the python process. Basically all my functions are decorated. See for example in wdoc, my advanced personal RAG system: https://github.com/thiswillbeyourgithub/wdoc/blob/main/wdoc/...

I recently built a semantic batching function for my RAG system [wdoc](https://github.com/thiswillbeyourgithub/wdoc/) that might be interesting to others. The system splits a corpus into chunks, finds relevant ones via embeddings, and answers questions for each chunk in parallel before aggregating the answers. To optimize performance and reduce LLM distraction, instead of aggregating answers two by two, it does batched aggregation. The key innovation is in the batching order - I implemented a [semantic_batching function](https://github.com/thiswillbeyourgithub/wdoc/blob/18bc52128f...) that uses hierarchical clustering on the embeddings and orders texts by leaf order. The implementation was straightforward, runs very fast and produces great results. The function is designed to be usable as a standalone tool for others to experiment with.

I've made wdoc just for that: https://github.com/thiswillbeyourgithub/WDoc I am a medical student with thousands of pdfs, various anki databases, video conferences, audio recordings, markdown notes etc. It can query into all of them and return extremely high quality output with sources to each original document. It's still in alpha though and there's only 0.5 user beside me that I know of so there are bugs that have yet to be found!

I'm a medical student with FOMO and who loves AI stuff. I couldn't keep up with my news so I made the perfect summarizer that goes through the thought process of the author : https://github.com/thiswillbeyourgithub/WDoc I needed an AI based system that go through my anki cards, but might as well make it able to read dozens of file formats. Now I can put entire medical youtube playlists, conferences, anki databases, hundreds of PDFs and ask a single question across all of them at once . It's both the same project

For my RAG projet [WDoc](https://github.com/thiswillbeyourgithub/WDoc/tree/dev) I use multiple pdf parser then use heuristics the keep the best one. The code is at https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2... And the heurstics are partly based on using fasttext to detecr languages : https://github.com/thiswillbeyourgithub/WDoc/blob/654c05c5b2... It's probably crap for tables but I don't want to rely on external parsers.