R2R

al async calls: - How do you consistently retry transient errors (like 503s from Azure's API) with proper backoff and jitter? - How do you stop hammering a service that's clearly down (circuit breaking) while automatically testing recovery? - How do you manage API rate limits across concurrent tasks without complex manual logic? - And importantly, how do you provide standardized, actionable error information to both your server and any clients depending on your API? Toller aims to solve this with a simple `@toller.task` decorator that wraps your `async` functions and adds: Rate Limiting:Async-safe token bucket-based limiter with adaptive throttling that responds to service feedback Retries: Customizable strategies (max attempts, delay, exponential backoff, jitter) on specific exceptions Circuit Breaker: Standard CLOSED/OPEN/HALF_OPEN states with configurable thresholds and recovery testing Transparent Error Handling: Rich context in exceptions that makes it clear what happened and what to do about it Unlike alternatives like Tenacity (focused on retries but lacking circuit breaking) or raw asyncio primitives (which require significant boilerplate), Toller provides an integrated solution specifically designed for modern async workflows, particularly those involving LLMs and other AI services where reliability is critical. A clear exception hierarchy (`TransientError`, `FatalError`, `MaxRetriesExceeded`, `OpenCircuitError`) aims to make error handling predictable. The goal is to take the "unruly" nature of distributed async calls and "lure" them into well-managed flows (hence the name, inspired by the Nova Scotia Duck Tolling Retriever!). It's async-native, lightweight, and I've tried to make it as straightforward as possible. I'd love to get your feedback, suggestions, or hear about any use cases you might have for it! Thanks!

servable RAG system built on top of a RESTful API that we shared back in February. What's the problem? Developers are struggling to build accurate, reliable RAG solutions. Popular tools like Langchain are complex and overly abstracted and lack crucial production features such as user/document management, observability, and a default API. There was a big thread about this a few days ago: Why we no longer use LangChain for building our AI agents (https://news.ycombinator.com/item?id=40739982) We experienced these challenges firsthand while building a large-scale semantic search engine, having users report numerous hallucinations and inaccuracies. This highlighted that search+RAG is a difficult problem. We're convinced that these missing features, and more, are essential to effectively monitor and improve such systems over time. Teams have been using R2R to develop custom AI agents with their own data, with applications ranging from B2B lead generation to research assistants. Best of all, the developer experience is much improved. For example, we have recently seen multiple teams use R2R to deploy a user-facing RAG engine for their application within a day. By day 2 some of these same teams were using their generated logs to tune the system with advanced features like hybrid search and HyDE. Here are a few examples of how R2R can outperform classic RAG with semantic search only: 1. “What were the UK's top exports in 2023?". R2R with hybrid search can identify documents mentioning "UK exports" and "2023", whereas semantic search finds related concepts like trade balance and economic reports. 2. "List all YC founders that worked at Google and now have an AI startup." Our knowledge graph feature allows R2R to understand relationships between employees and projects, answering a query that would be challenging for simple vector search. The built in observability and customizability of R2R helps you to tune and improve your system long after launching. Our plan is to keep the API ~fixed while we iterate on the internal system logic, making it easier for developers to trust R2R for production from day 1. We are currently working on the following: (1) Improve semantic chunking through third party providers or our own custom LLMs; (2) Training a custom model for knowledge graph triples extraction that will allow KG construction to be 10x more efficient. (This is in private beta, please reach out if interested!); (3) Ability to handle permissions at a more granular level than just a single user; (4) LLM-powered online evaluation of system performance + enhanced analytics and metrics. Getting started is easy. R2R is a lightweight repository that you can install locally with `pip install r2r`, or run with Docker. Check out our quickstart guide: https://r2r-docs.sciphi.ai/quickstart. Lastly, if it interests you, we are also working on a cloud solution at https://sciphi.ai. Thanks a lot for taking the time to read! The feedback from the first ShowHN was invaluable and gave us our direction for the last three months, so we'd love to hear any more comments you have!

nt; (3) after a (human) user comments, generates a side PR with the fix; and (4) after a (human) user merges the side PR and adds another commit, re-reviews the PR and approves it Here’s another example: https://github.com/SciPhi-AI/R2R/pull/350#pullrequestreview-..., where Ellipsis adds several comments with inline suggestions that were directly merged by the developer. You can configure Ellipsis in natural language to enforce custom rules, style guides, or conventions. For example, here’s how the `jxnl/instructor` repo uses natural language rules to make sure that docs are kept in sync: https://github.com/jxnl/instructor/blob/main/ellipsis.yaml#L..., and here’s an example PR that Ellipsis came up with based on those rules: https://github.com/jxnl/instructor/pull/346. Installing into your repo takes 2 clicks at https://www.ellipsis.dev. You do have to sign up to try it out because we need you to authorize our GitHub app to read your code. Don’t worry, your code is never stored or used to train models (https://docs.ellipsis.dev/security). We’d really appreciate your feedback, thoughts, and ideas!

onstrably not very good at retrieval This is very true - a short/medium term goal of mine is to integrate some more intelligent chunking approaches, ranging from Vikp's Surya to Reducto's proprietary model. I'm also interested in exploring what can be done from the pure software side. - the R in rag is really just "how can you do the best possible search for the given query". The approach here is so simple that it is definitely not the best possible search results. It's missing so many known techniques right now like: - Generate example queries that the chunk can answer and embed those to search against. - Parent document retrieval - so many newer better Rag techniques have been talked about and used that are better than chunk based - How do you differentiate "needs all source" vs "find in source" questions? Think: Summarize the entire pdf, vs a specific question like how long does it take for light to travel to the moon and back? You mentioned "Generate example queries", there is already an example that shows how to generate and search over synthetic queries w/ minor tweaks to the basic pipeline [https://github.com/SciPhi-AI/R2R/blob/main/examples/academy/...]. I think the other other approaches you outline are all worth investigating as well. There is definitely a tension we face between building and testing new experimental approaches vs. figuring out what features people need in production and implementing those. Just so you know where we are heading - we want to make sure all the features are there for easy experimentation, but we also want to provide value into production and beyond. As an example, we are currently working on robust task orchestration to accompany our pipeline abstractions to help with ingesting large quantities of data, as this has been a painpoint in our own experience and that of some of our early enterprise users.

tions common in other projects. Through conversations with numerous developers, we discovered that many were independently developing similar solutions. R2R distinguishes itself by adopting a straightforward approach to streamline the setup, monitoring, and upgrading of RAG systems. Specifically, it focuses on reducing unnecessary complexity and enhancing the visibility and tracking of system performance. The key parts of R2R include: an Ingestion Pipeline that transforms different data types (like json, txt, pdf, html) into 'Documents' ready for embedding. Next, the Embedding Pipeline takes text and turns it into vector embeddings through various processes (such as extracting text, transforming it, chunking, and embedding). Finally, the RAG Pipeline follows the steps of the embedding pipeline but adds an LLM provider to create text completions. R2R is currently in use at several companies building applications from B2B lead generation to educational tools for consumers. Our GitHub repo (https://github.com/SciPhi-AI/R2R) includes basic examples for application deployment and standalone use, demonstrating the framework's adaptability in a simple way. We’d love for you to give R2R a try, and welcome your feedback and comments as we refine and develop it further!