BricksLLM

sts etc.) * Building an AI dev platform with observability and monitoring * Building token usage based billing * Blocking or redacting requests that contains PII (https://github.com/bricks-cloud/BricksLLM/blob/main/cookbook/pii_detection.md) * Improving reliability via retries, failovers and API key rotations We believe that one underrated aspect of an LLM platform is the developer experience. The performance of a foundational model is important. However, LLMs will become less differentiated as the competition heats up among different providers. Therefore, a good developer experience is just as crucial in creating a competitive advantage. This is why I believe OpenAI has invested a lot into improving the DX of its platform. Now with BricksLLM, you can do the same to your own LLMs. The latency of the gateway is about 15ms on my M1 Chip MacBook pro. We have load tested our applications to be able to handle more than 1000 requests/s (https://github.com/bricks-cloud/BricksLLM/blob/main/blog/how-we-built-a-highly-sca…). Would love to hear your feedbacks and opinions! Open Source Repo: https://github.com/bricks-cloud/bricksllm Landing Page: https://trybricks.ai

first started building with OpenAI APIs, I was constantly worried about API keys being comprised since vanilla OpenAI API keys would grant you unlimited access to all of their models. There are stories of people losing thousands of dollars and the existence of a black market for stolen OpenAI API keys. This is why I started building a proxy for ourselves that allows for the creation of API keys with rate limits and cost controls. I built BricksLLM in Go since that was the language I used to build performative ads exchanges that scaled to thousands of requests per second at my previous job. A lot of developer tools in LLM ops are built with Python which I believe might be suboptimal in terms of performance and compute resource efficiency. One of the challenges building this platform is to get accurate token counts for different OpenAI and Anthropic models. LLM providers are not exactly transparent with the way how they count prompt and completion tokens. In addition to user input, OpenAI and Anthropic pad prompt inputs with additional instructions or phrases that contribute to the final token counts. For example, Anthropic's actual completion token consumption is consistently 4 more than the token count of the completion output. The latency of the gateway hovers around 50ms. Half of the latency comes from the tokenizer. If I start utilizing Go routines, might be able to lower the latency of the gateway to 30ms. BricksLLM is not an observability platform, but we do provide integration with Datadog so you can get more insights regarding what is going on inside the proxy. Compared to other tools in the LLMOps space, I believe that BricksLLM has the most comprehensive features when it comes to access control. Let me know what you guys think.