dataherald

Spider, but with a new set of hand-selected questions and queries grouped by query category. For an in-depth look into our process of creating this evaluation approach, see this. > Our testing procedure comprises the following steps. For each question/query pair: 1. We generate a SQL query (possibly from an LLM). 2. We run both the "gold" query and the generated query on their respective database to obtain 2 dataframes with the results. 3. We compare the 2 dataframes using an "exact" and a "subset" match. TODO add link to blogpost. 4. We log these alongside other metrics of interest (e.g. tokens used, latency) and aggregate the results for reporting - dataherald/services/engine/dataherald/tests/sql_generator/test_generator.py: https://github.com/Dataherald/dataherald/blob/main/services/...

and unstructured documents which are stored in a data store or a vector DB and injected if relevant - Allowing users to upload sample NL <> SQL pairs (golden SQL) which can be used in few shot prompting or to fine-tune an NL-to-SQL LLM for that specific dataset - Executing the SQL against the DB to get a few sample rows and recover from errors - Using an evaluator to assign a confidence score to the generated SQL The repo includes four services https://github.com/Dataherald/dataherald/tree/main/services: 1- Engine: The core service which includes the LLM agent, vector stores and DB connectors. 2- Admin Console: a NextJS front-end for configuring the engine and observability. 3- Enterprise Backend: Wraps the core engine, adding authentication, caching, and APIs for the frontend. 4- Slackbot: Integrate Dataherald directly into your Slack workflow for on-the-fly data exploration. Would love to hear from the community on building natural language interfaces to relational data. Anyone live in production without a human in the loop? Thoughts on how to improve performance without spending weeks on model training?

ssume the question is about Los Angeles, CA or would confirm the state with the question asker in a follow up. However, an LLM translates this to: select price from rent_prices where city=”Los Angeles” AND month=”05” AND year=”2023” Dataherald integrates with major data warehouses, including PostgreSQL, Databricks, Snowflake, BigQuery, and DuckDB. You can try it now free – no fees, no credit card, no sales pitches, just get the API key and get going. Let us know if it works for you, even your complex queries. (https://console.dataherald.ai/playground) While the open source version works just fine (https://github.com/Dataherald/dataherald), the hosted API might be a better fit for those looking for: (1) someone else to take care of infrastructure setup, (2) access to an Admin UI console where you can configure and monitor performance, and (3) ability to invite team members to a project. We're looking for feedback, particularly from anyone who can compare this performance to other NL-to-SQL products. Share your thoughts and join the conversation For more background on the release: https://www.dataherald.com/news/introducing-dhai

TX without getting columns to differentiate between the two. You can read more about the challenges of enterprise-level text-to-SQL in this blog post I wrote on the topic: https://medium.com/dataherald/why-enterprise-natural-languag... Dataherald comes with “batteries-included.” It has best-in-class implementations of core components, including, but not limited to: a state of the art NL-to-SQL agent, an LLM-based SQL-accuracy evaluator. The architecture is modular, allowing these components to be easily replaced. It’s easy to set up and use with major data warehouses. There is a “Context Store” where information (NL2SQL examples, schemas and table descriptions) is used for the LLM prompts to make the engine get better with usage. And we even made it fast! This version allows you to easily connect to PG, Databricks, BigQuery or Snowflake and set up an API for semantic interactions with your structured data. You can then add business and data context that are used for few-shot prompting by the engine. The NL-to-SQL agent in this open source release was developed by our own Mohammadreza Pourreza, whose DIN-SQL algorithm is currently top of the Spider (https://yale-lily.github.io/spider) and Bird (https://bird-bench.github.io/) NL 2 SQL benchmarks. This agent has outperformed the Langchain SQLAgent anywhere from 12%-250%.5x (depending on the provided context) in our own internal benchmarking while being only ~15s slower on average. Needless to say, this is an early release and the codebase is under swift development. We would love for you to try it out and give us your feedback! And if you are interested in contributing, we’d love to hear from you!