harmony

Because nobody has figured out how to make that work 100% reliably yet. The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#s... Then you train your model to take those tokens into account. Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277 > We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label

Take a look at the harmony repo which specifies the internal OpenAI format - the effort level is specified in the context after the <|start|> tag - https://github.com/openai/harmony Note that inference libs also have parsers that put hard limits on reasoning tokens with separate counters (similar to how you can put a limit on token generation per completion versus waiting for an <eos>). For that, take a look at vllm reasoning docs.

Since GPS-OSS there is also the Harmony response format (https://github.com/openai/harmony) that instead of just having a system/assistant/user split in the roles, instead have system/developer/user/assistant/tool, and it seems to do a lot better at actually preventing users from controlling the LLM too much. The hierarchy basically becomes "system > developer > user > assistant > tool" with this.

> GPT-OSS is not much of a snowflake Yeah, according to the architecture it doesn't seem like a snowflake, but they also decided to invent a new prompting/conversation format (https://github.com/openai/harmony) which definitely makes it a bit of a snowflake today, can't just use what worked a couple of days ago, but everyone needs to add proper support for it.

Related, link to the GH repo: https://github.com/openai/harmony

Inference in Python uses harmony [1] (for request and response format) which is written in Rust with Python bindings. Another OpenAI's Rust library is tiktoken [2], used for all tokenization and detokenization. OpenAI Codex [3] is also written in Rust. It looks like OpenAI is increasingly adopting Rust (at least for inference). [1] https://github.com/openai/harmony [2] https://github.com/openai/tiktoken [3] https://github.com/openai/codex

What a day! Models aside, the Harmony Response Format[1] also seems pretty interesting and I wonder how much of an impact it might have in performance of these models. [1] https://github.com/openai/harmony