morphik-core

rong page, and still couldn’t answer the question. We wrote up the full story with screenshots here: https://docs.morphik.ai/blogs/gpt-vs-morphik-multimodal. We got frustrated enough to try fixing it ourselves. We built Morphik to do multimodal retrieval over documents like PDFs, where images and diagrams matter as much as the text. To do this, we use Colpali-style embeddings, which treat each document page as an image and generate multi-vector representations. These embeddings capture layout, typography, and visual context, allowing retrieval to get a whole table or schematic, not just nearby tokens. Along with vector search, this could now retrieve exact pages with relevant diagrams and pass them as images to the LLM to get relevant answers. It’s able to answer the question with an 8B llama 3.1 vision running locally! Early pharma testers hit our system with queries like "Which EGFR inhibitors at 50 mg showed ≥ 30% tumor reduction?" We correctly returned the right tables and plots, but still hit a bottleneck, we weren’t able to join the dots across multiple reports. So we built a knowledge graph: we tag entities in both text and images, normalize synonyms (Erlotinib → EGFR inhibitor), infer relations (e.g. administered_at, yields_reduction), and stitch everything into a graph. Now a single query could traverse that graph across documents and surface a coherent, cross‑document answer along with the correct pages as images. To illustrate that, and just for fun, we built a graph of 100 Paul Graham’s essays here: https://pggraph.streamlit.app/ You can search for various nodes, (eg. startup, sam altman, paul graham and see corresponding connections). In our system, we create graphs and store the relevant text chunks along with the entities, so on querying, we can extract the relevant entity, do a search on the graph and pull in the text chunks of all connected nodes, improving cross document queries. For longer or multi-turn queries, we added persistent KV caching, which stores intermediate key-value states from transformer attention layers. Instead of recomputing attention from scratch every time, we reuse prior layers, speeding up repeated queries and letting us handle much longer context windows. We’re open‑source under the MIT Expat license: https://github.com/morphik-org/morphik-core Would love to hear your RAG horror stories, what worked, what didn’t and any feedback on Morphik. We’re here for it.

ographics. Something that can help solve this is directly embedding the document as a list of images, and performing retrieval over that - getting the closest images to the query, and feeding the LLM exactly those images. This helps reduce the amount of tokens an LLM consumes while also increasing the visual reasoning ability of the model. We've implemented a one-line solution that does exactly this with Morphik. You can check out the specifics in the attached blog, or get started with it through our quick start guide: https://docs.morphik.ai/getting-started Here is an example ingestion pathway: ``` from databridge import DataBridge db = DataBridge() db.ingest_file("report_with_images_and_graphs.pdf", use_colpali=True) ``` And here is an example query pathway: ``` db.query("At what time-step did we see the highest GDP growth rate?", use_colpali=True) ``` Would love to hear your thoughts!

_file(file="/path/file.pdf", filename="report2025", use_colpali=True, rules=[MetadataExtraction(schema=json_object)]) This single command embeds text, visual elements, and metadata into one vector space. This article explains Colpali vs traditional parsing pipelines better than I can: https://medium.com/@shashankvats/colpali-explained-bridging-.... Or for those more technically inclined: https://arxiv.org/abs/2407.01449 We’re also currently working on an MCP server integration so tools like Claude Desktop can directly benefit from this enhanced context. Would love your thoughts and any feedback. Docs: https://databridge.gitbook.io/databridge-docs

=True` in our python SDK while ingesting or retrieving documents. Our long term goal is to make state of the art research in retrieval be as accessible for production use cases as possible, and integrating ColPali is an initial step towards that goal. If there's research that you think is compelling, but haven't been able to integrate into production, let us know: we'd be happy to help. We really appreciate the honest feedback the HN community provides, and so we'd love to hear from you!

community, so I'd love your feedback about what features you're currently missing in RAG pipelines, whether specialized parsing (e.g., for medical docs, legal texts, or multimedia) is something you'd want, what your ideal RAG workflow looks like, and what some must-haves are. Thanks for checking out DataBridge, and feel free to open issues or PRs on GitHub if you have ideas, requests, or want to help shape the next set of features. If this is helpful, I'd really appreciate it if you could give it a star on GitHub! Looking forward to hearing your thoughts! Happy building!