unstructured

I've been thinking a lot about how to accomplish various RAG things in Elixir (for LLM applications). PDF is one of the missing pieces, so glad to see work here. The really tricky part is not just parsing out the text (you can just call the pdftotext unix command line utility for that), but accurately pulling out things like complex tables, etc in a way that could be chunked/post processed in a useful way. I'd love to see something like Unstructured or Marker but in Rust (i.e., fast) that Elixir could NIF out to it. And maybe some kind of hybrid system that uses open llm models with vision capabilities. Ref: - https://github.com/Unstructured-IO/unstructured - https://github.com/VikParuchuri/marker

Depends on how fuzzy your definition of template is. If it's really structured, a simpler algorithm might work better. Otherwise maybe try https://github.com/Unstructured-IO/unstructured

Be careful with unstructured: https://github.com/Unstructured-IO/unstructured/blob/d11c70c... from: https://github.com/open-webui/open-webui/issues/687

I’ve been looking at this https://freeling-user-manual.readthedocs.io/en/v4.2/modules/... at the freeling library in general, also spaCy and NLTK. The chunking algorithms being used in the likes of LangChain are remarkably bad surprisingly. There is also https://github.com/Unstructured-IO/unstructured But I don’t like it, can’t explain why yet. My intuition is that 1st step is clean sentences and paragraphs and titles/labels/headers. Then probably an LLM can handle outlining and table of contents generation using a stripped down list of objects in the text. BRIO/BERT summarization could also have a role of some type. Those are my ideas so far.