cocoindex

nt through pivoting hell. In all the projects I’ve built, data still sits in the center of the problem. I need to write custom logic to prepare data for AI and I need the data to stay fresh for AI to generate effective decisions or recommendations. So I started CocoIndex. Getting started: https://cocoindex.io/docs/getting_started/quickstart Examples: https://github.com/cocoindex-io/cocoindex/tree/main/examples If you prefer to read: https://cocoindex.io/blogs/tags/examples/ Would love to learn your thoughts. Thanks! Linghua

bind to different languages. I’ve made a few tutorials and new projects since last launch, with different use cases: - https://www.youtube.com/@cocoindex-io - https://cocoindex.io/blogs/tags/examples Previously, I’ve worked at Google on projects like search indexing and ETL infra for 8 years. After I left Google last year, I built various projects and went through pivoting hell. In all the projects I’ve built, data still sits in the center of the problem and I find myself focusing on building data infra other than the business logic I need for data transformation. The current prepackaged RAG-as-service doesn't serve my needs, because I need to choose a different strategy for the context, and I also need deduplication, clustering (items are related), and other custom features that are commonly needed. That’s where CocoIndex starts. A simple philosophy behind it - data transformation is similar to formulas in spreadsheets. The ground of truth is at the source data, and all the steps to transform, and final target store are derived data, and should be reactive based on the source change. If you use CocoIndex, you only need to worry about defining transformations like formulas. Data flow paradigm came in as an immediate choice. because there’s no side effect, lineage and observability just come out of the box. Incremental processing - If you are a data expert, an analogy would be a materialized view beyond SQL. The framework tracks pipeline states in database (Postgres) and only re-processes necessary portions. When data has changed, the framework handles the change data capture comprehensively and combines the mechanism for push and pull. Then clear stale derived data/versions and re-index data based on tracking data/logic changes or data TTL settings. There’s lots of edge cases to do it right, for example, when a row is referenced in other places, and the row changes. These should be handled at the level of the framework. At the compute engine level - the framework should consider the multiple processes and concurrent updates. It should consider how to resume existing states from terminated execution. In the end, we want to build a framework that is easy to build with exceptional velocity, but scalable and robust in production. Standardized the interface throughout the data flow - really easy to plugin custom logic like LEGO; with a variety of native built-in components. One example is that it takes a few lines to switch among Qdrant, Postgres, Neo4j. CocoIndex is licensed under Apache 2.0 https://github.com/cocoindex-io/cocoindex Getting started: https://cocoindex.io/docs/getting_started/quickstart I have rolled out over 25 releases since last HN launch and it has significantly improved in all aspects, especially supporting property graphs (Neo4j, Kuzu), supporting queue based CDC (AWS S3, SQS) and lots of infra updates including CLI, resilience and error handlings. Excited to learn your thoughts, and thank you so much! Linghua