guidance

Guidance is an efficient programming paradigm for steering language models. With Guidance, you can control how output is structured and get high-quality output for your use case—while reducing latency and cost vs. conventional prompting or fine-tuning. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditionals, loops, tool use) and generation seamlessly. Install Features Install Guidance is available through PyPI and supports a variety of backends (Transformers, llama.cpp, OpenAI, etc.). If you already have the backend required for your model, you can simply run pip install guidance Features A Pythonic interface for language models When using Guidance, you can work with large language models using common Python idioms: from guidance import system, user, assistant, gen from guidance.models import Transformers # Could also do LlamaCpp or many other models phi_lm = Transformers("microsoft/Phi-4-mini-instruct") # Model objects are immutable, so this is a copy lm = phi_lm with system(): lm += "You are a helpful assistant" with user(): lm += "Hello. What is your name?" with assistant(): lm += gen(max_tokens=20) print(lm) If run at the command line, this will produce output like: <|system|>You are a helpful assistant<|end|><|user|>Hello. What is your name?<|end|><|assistant|>I am Phi, an AI developed by Microsoft. How can I help you today? However, if running in a Jupyter notebook, then Guidance provides a widget for a richer user experience: With Guidance, it's really easy to capture generated text: # Get a new copy of the Model lm = phi_lm with system(): lm += "You are a helpful assistant" with user(): lm += "Hello. What is your name?" with assistant(): lm += gen(name="lm_response", max_tokens=20) print(f"{lm['lm_response']=}") lm['lm_response']='I am Phi, an AI developed by Microsoft. How can I help you today?' Guarantee output syntax with constrained generation Guidance provides an easy to use, yet immensely powerful syntax for constraining the output of a language model. For example, a gen() call can be constrained to match a regular expression: lm = phi_lm with system(): lm += "You are a teenager" with user(): lm += "How old are you?" with assistant(): lm += gen("lm_age", regex=r"\d+", temperature=0.8) print(f"The language model is {lm['lm_age']} years old") The language model is 13 years old Often, we know that the output has to be an item from a list we know in advance. Guidance provides a select() function for this scenario: from guidance import select lm = phi_lm with system(): lm += "You are a geography expert" with user(): lm += """What is the capital of Sweden? Answer with the correct letter. A) Helsinki B) Reykjavík C) Stockholm D) Oslo """ with assistant(): lm += select(["A", "B", "C", "D"], name="model_selection") print(f"The model selected {lm['model_selection']}") The model selected C The constraint system offered by Guidance is extremely powerful. It can ensure that the output conforms to any context free grammar (so long as the backend LLM has full support for Guidance). More on this below. Debug grammars offline (no model API calls) When iterating on constraints, you can validate candidate strings locally and test a full run with the Mock model. from guidance import gen from guidance.models import Mock grammar = "expr=" + gen(regex=r"\d+([+*]\d+)*", name="expr") # 1) Validate strings directly against the grammar assert grammar.match("expr=12+7*3") is not None assert grammar.match("expr=12+*3") is None # 2) Run the same grammar with a local mock model lm = Mock(b"<s>expr=12+7*3") lm += grammar print(lm["expr"]) # 12+7*3 Create your own Guidance functions With Guidance, you can create your own Guidance functions which can interact with language models. These are marked using the @guidance decorator. Suppose we wanted to answer lots of multiple choice questions. We could do something like the following: import guidance from guidance.models import Model ASCII_OFFSET = ord("a") @guidance def zero_shot_multiple_choice( language_model: Model, question: str, choices: list[str], ): with user(): language_model += question + "\n" for i, choice in enumerate(choices): language_model += f"{chr(i+ASCII_OFFSET)} : {choice}\n" with assistant(): language_model += select( [chr(i + ASCII_OFFSET) for i in range(len(choices))], name="string_choice" ) return language_model Now, define some questions: questions = [ { "question" : "Which state has the northernmost capital?", "choices" : [ "New South Wales", "Northern Territory", "Queensland", "South Australia", "Tasmania", "Victoria", "Western Australia", ], "answer" : 1, }, { "question" : "Which of the following is venomous?", "choices" : [ "Kangaroo", "Koala Bear", "Platypus", ], "answer" : 2, } ] We can use our decorated function like gen() or select(). The language_model argument will be filled in for us automatically: lm = phi_lm with system(): lm += "You are a student taking a multiple choice test." for mcq in questions: lm_temp = lm + zero_shot_multiple_choice(question=mcq["question"], choices=mcq["choices"]) converted_answer = ord(lm_temp["string_choice"]) - ASCII_OFFSET print(lm_temp) print(f"LM Answer: {converted_answer}, Correct Answer: {mcq['answer']}") <|system|>You are a student taking a multiple choice test.<|end|><|user|>Which state has the northernmost capital? a : New South Wales b : Northern Territory c : Queensland d : South Australia e : Tasmania f : Victoria g : Western Australia <|end|><|assistant|>b LM Answer: 1, Correct Answer: 1 <|system|>You are a student taking a mu