sql-eval

Benchmarks suggest otherwise. Toqan's sql benchmark shows other models way up in the ranking compared to gpt-4o [1] Open Weight models specifically fine-tuned on sql generation and modification also rank pretty well compared to SOTA proprietary models. If you want to eval alternative models, check out sqleval [2] 1 https://prollm.toqan.ai/leaderboard/stack-unseen?type=concep... 2 https://github.com/defog-ai/sql-eval

/? awesome "sql" llm site:github.com https://www.google.com/search?q=awesome+%22sql%22+llm+site%3... : - awesome-Text2SQL: https://github.com/eosphoros-ai/Awesome-Text2SQL : > Curated tutorials and resources for Large Language Models, Text2SQL, Text2DSL、Text2API、Text2Vis and more. - Awesome-code-llm > Benchmarks > Text to SQL: https://github.com/codefuse-ai/Awesome-Code-LLM#text-to-sql - underlines/awesome-ml//llm-tools.md > RAG > OpenAI > dataherald,: https://github.com/underlines/awesome-ml/blob/master/llm-too... - underlines/awesome-ml//llm-tools.md > Benchmarking > Benchmark Suites, Leaderboards: https://github.com/underlines/awesome-ml/blob/master/llm-too... : - sql-eval: https://github.com/defog-ai/sql-eval : > This repository contains the code that Defog uses for the evaluation of generated SQL. It's based off the schema from the

Can you give me an example of the sort of thing you're talking about? I've been using Defog's sql-eval a little bit, but I'd be interested in knowing more about its shortcomings when evaluating these systems. https://github.com/defog-ai/sql-eval

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%. What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? As a programmer who always has to look stuff up when I SQL, I could definitely see asking something like this for a first draft of a query but it seems like I'm slightly better off asking the bigger models in these one-off cases (and I can run a 15b easily on my 64GB m1). If I'm in a corporate setting I'm not going to leak my schema into OpenAI's training data and there are definitely times when I'd want to run queries offline. Small/local models are great when you want to do a ton of queries (save $$). A mini data scientist that could be queried by non-technical folks would be awesome but I wonder if there's

Is this the data that was used for fine tuning? https://github.com/defog-ai/sql-eval/blob/main/data/question...