role-model

%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%. Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated. First principles of model routing: https://try.works/first-principles-of-model-routing role-model router and protocol: https://github.com/try-works/role-model note: edited to keep the answer to the below message clearer

cache hit rate then becomes 2k/200k = 99% instead of 99.5%. is it worth it? depends on the models in your pool. If you have GPT 5.6 and DeepSeek, it's worth it because the cost difference is vast.

ap. Since there was another model router post yesterday where people discussed the basics of routing, I will focus on discussing some of the interesting learnings I've made building and testing this: 1. Model routing is essentially trying to predict the future: which model will perform optimally (based on criteria defined by the user) on this request? 2. After you have routed the request, you want to evaluate if it was the right decision or if some other model would have performed better 3. You also realize that having the router assess difficulty (among other things) to make decisions by itself is far from ideal - we'd prefer to have the consumer application work with the router to define what the request needs 4. You also realize that it becomes much easier, decisions become much accurate, and the outcomes of routing becomes more impactful when there is more of a distinction between models For point 2, I will be launching evals that you can run locally to benchmark models in your pool on the same requests. The outcomes here can then be used for point 1, as input when routing new requests. For point 3, I've built the pi-role-model package for Pi, which lets the Pi agent inject role_model.intent metadata including difficulty, preferred roles or even specfic model ids, required capabilities (say tool use or image input) and so on. You should be able to customized this further in Pi, and route in additional ways by changing metadata. This is why I've also built the role-model routing protocol. For point 4, what model routing really does as a second order effect is create a market for specialized models - models that may or may not be smaller, could be cheaper or more expensive, may be locally runnable. It makes little sense to route between two frontier models (GPT 5.5 and Opus 4.8); it makes more sense to route between models where one of the factors of quality, speed, cost is a multiple of the other candidate models, and it makes even more sense to have specialized domain models: code, prose, math and science, visuals and so on. It is at this stage model routing becomes really valuable. While role-model has a reference runtime that I'm continuously building out (there's lots to do to improve routing, as well as give users more granular control over routing decisions, and also ways to improve cross-model caching and also add techniques like FastContext), the ultimate goal of role-model is for there to be a standard protocol for inference requests that is used by consumer applications, so that the provider, be it a router middleware or an inference provider, will be able to route to a model that strikes the best balance between cost, speed and quality and also respects user choices, and even lets the user control these preferences to use local models for some tasks and allow cloud for others. Links: [0] role-model - the case for a model routing protocol: https://try.works/role-model-the-case-for-a-model-routing-pr... [1] GitHub: https://github.com/try-works/role-model [2] Docs: https://role-model.dev/