outlines

trumps voice wrong https://developers.openai.com/api/docs/guides/structured-out... https://github.com/guidance-ai/guidance https://github.com/noamgat/lm-format-enforcer https://github.com/mlc-ai/xgrammar https://github.com/dottxt-ai/outlines

How does this differ from dottxt's Outlines[0] on the technical level? Are you using some JSON grammar to force the LM head distribution to follow it? [0]: https://github.com/dottxt-ai/outlines

This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested: Some libraries: - Outlines, a nice library for structured generation - https://github.com/dottxt-ai/outlines - Guidance (already covered by FlyingLawnmower in this thread), another nice library - https://github.com/guidance-ai/guidance - XGrammar, a less-featureful but really well optimized constrained generation library - https://github.com/mlc-ai/xgrammar - This one has a lot of cool technical aspects that make it an interesting project Some papers: - Efficient Guided Generation for Large Language Models - By the outlines authors, probably the first real LLM constrained generation paper - https://arxiv.org/abs/2307.09702 - Automata-based constraints for language model dec

Use one of these structured output libraries: https://github.com/outlines-dev/outlines https://github.com/jxnl/instructor https://github.com/guardrails-ai/guardrails https://www.askmarvin.ai/docs/text/transformation/ Some of them allow a JSON schema, others a Pydantic model (which you can transform to/from JSON).

Any of the "design patterns" listed in the article will have a ton of popular open source implementations. For structured generation, I think outlines is a particularly cool library, especially if you want to poke around at how constrained decoding works under the hood: https://github.com/dottxt-ai/outlines

`outlines` (https://github.com/dottxt-ai/outlines) is very good and supported by vLLM as a backend structured output provider (https://docs.vllm.ai/en/v0.8.2/features/structured_outputs.h...) for both local and remote LLMs. vLLM is probably the best open source tooling for the inference side right now.

I assume they're doing "Structured Generation" or "Guided generation", which has been possible for a while if you control the LLM itself e.g. running an OSS model, e.g. [0][1]. It's cool to see a major API provider offer it, though. The basic idea is: at each auto-regressive step (each token generation), instead of letting the model generate a probability distribution over "all tokens in the entire vocab it's ever seen" (the default), only allow the model to generate a probability distribution over "this specific set of tokens I provide". And that set can change from one sampling set to the next, according to a given grammar. E.g. if you're using a JSON grammar, and you've just generated a `{`, you can provide the model a choice of only which tokens are valid JSON immediately after a `{`, etc. [0] https://github.com/dottxt-ai/outlines [1]

If I understand it correctly, that's a valid concern but the way structured generation library like outlines[1] work is that they can generate multiple variants of the inference (which they call beam search). One beam could be "This is a way to solv-". With no obvious "good" next token. Another beam could be "This way is solv-". With "ing" as the obvious next token. It will select the best beam for the output. [1]:https://github.com/dottxt-ai/outlines

That could now be achieved by precomputing all valid moves and using outlines[0] or Structured Outputs[1] to constrain the output. [0]: https://github.com/dottxt-ai/outlines [1]: https://openai.com/index/introducing-structured-outputs-in-t...

Have you looked into structured generation with a library like outlines? https://github.com/dottxt-ai/outlines

Wow, this is cool-- first time I've seen generating well formed output has been made easy, the use of Pydantic is clever/easy... Which has lead me to the prior art: using same Pydantic API to generate structured output from local LLMs https://github.com/outlines-dev/outlines?tab=readme-ov-file#...