clownfish

https://github.com/newhouseb/clownfish Which I've been using for a while now, also restricts the sampling space to force correct generation, but does so as the result of a different process than yours.

I'm not sure how this is different than: https://github.com/1rgs/jsonformer or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or https://github.com/ggerganov/llama.cpp/pull/1773 or https://github.com/Shopify/torch-grammar Overall there are a ton of these logit based guidance systems, the reason they don't get tons of traction is the SOTA models are behind REST APIs that don't enable this fine-grained approach. Those models perform so much better that people generally settle for just re-requesting until they get the correct format (and with GPT-4 that ends up being a fairly rare occurrence in my experience)

But only for the whole generation. So if you want to constrain things one token at a time (as you would to force things to follow a grammar) you have to make fresh calls and only request one token which makes things more or less impractical if you want true guarantees. A few months ago I built this anyway to suss out how much more expensive it was [1] [1] https://github.com/newhouseb/clownfish#so-how-do-i-use-this-...

As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation: The model can choose to call a function; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may generate invalid JSON or hallucinate parameters). So sadly, it is just fine tuning. There's no hard biasing applied :(. You were so close, but so far OpenAI! [1] https://github.com/newhouseb/clownfish [2] https://platform.openai.com/docs/guides/gpt/function-calling

The interface makes it look simple, but under the hood it follows a similar approach to jsonformer/clownfish [1] passing control of generation back and forth between a slow LLM and relatively fast python Let's say you're halfway through a generation of a json blob with a name field and a job field and have already generated { "name": "bob" At this point, guidance will take over generation control from the model to generate the next text { "name": "bob", "job": If the model had generated that, you'd be waiting 70 ms per token (informal benchmark on my M2 air). A comma, followed by a newline, followed by "job": is 6 tokens, or 420ms. But since guidance took over, you save all that time. Then guidance passes control back to the model for generating the next field value. { "name": "bob", "job": "programmer" progra

I agree, it looks great. A couple similar projects you might find interesting: - https://github.com/newhouseb/clownfish - https://github.com/r2d4/rellm The first one is JSON only and the second one uses regular expressions, but they both take the same "logit masking" approach as the project GP linked to.

I built a similar thing to Grant's work a couple months ago and prototyped what this would look like against OpenAI's APIs [1]. TL;DR is that depending on how confusing your schema is, you might expect up to 5-10x the token usage for a particular prompt but better prompting can definitely reduce this significantly. [1] https://github.com/newhouseb/clownfish#so-how-do-i-use-this-...

https://github.com/newhouseb/clownfish Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema

This seems extremely similar to https://github.com/newhouseb/clownfish

Oh nice! I built a similar system a few weeks ago: https://github.com/newhouseb/clownfish I think the main differentiating factor here is that this is better if you have a simpler JSON schema without enums or oneOf constraints. If you do have these constraints, i.e. let's say you wanted an array of different types that represented a items on a menu { kind: pizza, toppings: [pepperoni] } or { kind: ice_cream, flavor: vanilla | strawberry } then you would need something more sophisticated like clownfish that can ask the LLM to pick specific properties (and an ability to do some backtracking so you can do proper beam search). For completeness, another common approach can be found here: https://github.com/ShreyaR/guardrails which essentially boils down to "provide the schema in the prompt and ask the LLM to correct things if it fails to get t

You're basically describing https://github.com/newhouseb/clownfish/, except there it tries to validate the output against JSON schema on the fly.

There are a couple different approaches: - Use multi-shot prompting with something like guardrails to try prompting a commercial model until it works. [1] - Use a local model with a final layer that steers token selection towards syntactically valid tokens [2] [1] https://github.com/ShreyaR/guardrails [2] "Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema" @ https://github.com/newhouseb/clownfish (full disclosure: this is my work)

And on the other end, there's https://github.com/newhouseb/clownfish to force the model to produce structured output.

It doesn't help with repetition, but when it comes to force structure on the output data, this approach looks interesting: https://github.com/newhouseb/clownfish TL;DR: it exploits the fact that the model returns probabilities for all the possible following tokens to enforce a JSON schema on the output as it is produced, backtracking as needed.