langtrace

nk about mostly while we are building Langtrace as well.

ain you are back to square one. This is a monolithic and unsustainable way to optimize. This is what DSPy tries to solve. DSPy is primarily two things: 1. Separate programming from prompt engineering using signatures and give greater control to program language models using traditional programming techniques. 2. Incorporate best practice prompting techniques in the framework to optimize the LMs response. With DSPy, you can implement and iterate on improving your app additional tools at your disposal. First, you can separate your prompts and write a signature instead. A signature looks like this “context, question -> answer”. This signature gets “compiled” into an actual prompt. In addition, you can also break this problem into 2 modules, 1. Retrieval module 2. Generation module Modules are classes that that take a signature with inputs and generate an output and you can start iterating on each module separately. Now, with the retrieval module, you can do a an iterative query generation & context retrieval before passing it to the generation module. This is a technique used to improve RAG performance by building up additional context by incrementally asking questions based on retrieved context and using the generate questions to retrieve additional context. With DSPy, this becomes straightforward because, DSPy exposes constructs like - Settings where you can define the number of hops - Signature like “context, question -> query” that gets compiled into a proper prompt - Optimizers to incrementally append retrieved contexts to the prompt automatically by the framework Coming to the answer generation module, it’s not atypical to define a set of rules while prompt engineering. DSPy solves this using run time assertions with backtracking where in, you can define an assertion instead of a rule and DSPy with automatically backtrack and improve the prompt if the assertion fails. DSPy will automatically back track and prefix the prompt to include additional rules to optimize the prompt further. Obviously, you can do this with prompt engineering & program composition. But, where DSPy really shines is when you are optimizing a large pipeline that is prone to brittleness with prompt engineering techniques. DSPy helps you compose the pipeline into isolated set of modules & lets you apply best practice prompting techniques without having to think much about the prompt. There are two implementations of the framework today. - Python - https://github.com/stanfordnlp/dspy - Typescript - https://github.com/dosco/llm-client/ One specific feature I would like to see is to expose the prompt templates as config and give greater control to the developers in incorporating our “versions” of templates for CoT, Fewshots etc. This will also help with the adoption of DSPy to languages other than English. Finally, I was able to understand what DSPy is adding to its prompts and what sort of optimizations it is doing by using Langtrace, an open source tracing tool i am building that can be setup with just 2 lines of code. https://github.com/Scale3-Labs/langtrace

observability client, but will also enable developers to develop tooling for any custom needs, such as capturing datasets, prompts, evaluations etc. A note on what we have built so far: [1] We have a Python and a TypeScript SDK and we have broken down the support for the LLM layer into 3 groups, LLMs, Frameworks and VectorDBs. Our SDKs are open telemetry compatible, can be installed and used independently and we also provide an option to pass custom exporters to export the traces and spans to any observability tool of your choice. [2] An observability client that is hyper optimized for solving the unique pain points and challenges that come with LLM based apps like Evaluations, prompt iteration, datasets etc. We are SOC2 compliant and the client can also be self hosted if you have strict data privacy and protection requirements. [3] Both the SDK and the client are fully open source. We are leaning on the community to try it out and provide us with feedback. A note about OpenTelemetry semantic conventions for LLMs - we would like to converge on standard names for trace attributes that follow the OTEL rules and are looking for feedback from experts here - https://github.com/Scale3-Labs/langtrace/discussions/71 We recognize that this project is early and there is a lot of room for improvement. We would love to hear your thoughts and feedback. Links: [1] https://langtrace.ai/ [2] https://github.com/Scale3-Labs/langtrace [3] https://docs.langtrace.ai/introduction [4] https://langtrace.ai/blog/why-you-need-opentelemetry-based-o...

itch their observability client, but will also enable developers to develop tooling for any custom needs such as capturing datasets, prompts, evaluations etc. A quick note on what we have built so far: [1] We have a Python and a TypeScript SDK and we have broken down the support for the LLM layer into 3 groups, LLMs, Frameworks and VectorDBs. Our SDKs are open telemetry compatible, can be installed and used independently and we also provide an option to pass custom exporters to export the traces and spans to any observability tool of your choice. [2] An observability client that is hyper optimized for solving the unique pain points and challenges that come with LLM based apps like Evaluations, prompt iteration, datasets etc. We are SOC2 compliant and the client can also be self hosted if you have strict data privacy and protection requirements. [3] Both the SDK and the client are fully open source. We are leaning on the community to try it out and provide feedback. A note about Open Telemetry semantic conventions for LLMs - We would like to converge on standard names for trace attributes that follow the OTEL rules and are looking for feedback from experts here - https://github.com/Scale3-Labs/langtrace/discussions/71 We recognize that this project is early and there is a lot of room for improvement. Would love to hear your thoughts and feedback. Thanks! Links: [1] https://langtrace.ai/ [2] https://github.com/Scale3-Labs/langtrace [3] https://docs.langtrace.ai/introduction

itch their observability client, but will also enable developers to develop tooling for any custom needs such as capturing datasets, prompts, evaluations etc. A quick note on what we have built so far: [1] We have a Python and a TypeScript SDK and we have broken down the support for the LLM layer into 3 groups, LLMs, Frameworks and VectorDBs. Our SDKs are open telemetry compatible, can be installed and used independently and we also provide an option to pass custom exporters to export the traces and spans to any observability tool of your choice. [2] An observability client that is hyper optimized for solving the unique pain points and challenges that come with LLM based apps like Evaluations, prompt iteration, datasets etc. We are SOC2 compliant and the client can also be self hosted if you have strict data privacy and protection requirements. [3] Both the SDK and the client are fully open source. We are leaning on the community to try it out and provide feedback. A note about Open Telemetry semantic conventions for LLMs - We would like to converge on standard names for trace attributes that follow the OTEL rules and are looking for feedback from experts here - https://github.com/Scale3-Labs/langtrace/discussions/71 We recognize that this project is early and there is a lot of room for improvement. Would love to hear your thoughts and feedback. Thanks! Links: [1] https://langtrace.ai/ [2] https://github.com/Scale3-Labs/langtrace [3] https://docs.langtrace.ai/introduction [4] https://langtrace.ai/blog/why-you-need-opentelemetry-based-o...

aving to switch their observability client, but will also enable developers to develop tooling for any custom needs, such as capturing datasets, prompts, evaluations etc. A note on what we have built so far: [1] We have a Python and a TypeScript SDK and we have broken down the support for the LLM layer into 3 groups, LLMs, Frameworks and VectorDBs. Our SDKs are open telemetry compatible, can be installed and used independently and we also provide an option to pass custom exporters to export the traces and spans to any observability tool of your choice. [2] An observability client that is hyper optimized for solving the unique pain points and challenges that come with LLM based apps like Evaluations, prompt iteration, datasets etc. We are SOC2 compliant and the client can also be self hosted if you have strict data privacy and protection requirements. [3] Both the SDK and the client are fully open source. We are leaning on the community to try it out and provide us with feedback. A note about OpenTelemetry semantic conventions for LLMs - we would like to converge on standard names for trace attributes that follow the OTEL rules and are looking for feedback from experts here - https://github.com/Scale3-Labs/langtrace/discussions/71 We recognize that this project is early and there is a lot of room for improvement. We would love to hear your thoughts and feedback. Thanks! Links: [1] https://langtrace.ai/ [2] https://github.com/Scale3-Labs/langtrace [3] https://docs.langtrace.ai/introduction [4] https://langtrace.ai/blog/why-you-need-opentelemetry-based-o...