pipecat

I think we're close to the point where local models can make this economically viable as an alternative to hands off XAI/OAI/ANT You need rapid good enough rapid turn response + rapid good enough TTS in order to make it economically for someone who's not blending it into a more compute heavy pricing model Have you looked into it? Are you familiar with Pipecat https://github.com/pipecat-ai/pipecat - they put out interesting demos frequently

if anyone is looking to get into this. pipecat is a great open-source repo and community. https://github.com/pipecat-ai/pipecat

pipecat is the best! (imo) https://github.com/pipecat-ai/pipecat

A debugger for Pipecat: https://github.com/pipecat-ai/pipecat With Whisker you can: - View a live graph of your pipeline - Watch frame processors flash in real time - Select a processor to inspect its frames - Filter frames by name - Select a frame to trace its full path

I recommend Daily.co + Pipecat! https://www.daily.co/ https://github.com/pipecat-ai/pipecat

My plan is that Espressif’s WebRTC code[0] will hook up to pipe at [1] that gets you the freedom to do whatever you want. The design of OpenAI + WebRTC was to lean on WebRTC as much as possible to make it easier for users. [0] https://github.com/espressif/esp-webrtc-solution [1] https://github.com/pipecat-ai/pipecat

Heya! I saw there was a mention of content moderation when the author discussed https://github.com/pipecat-ai/pipecat But when I went to the github repo, I didn't see anything about that. I'm loosely related to the content moderation space through my employer, so wanted to learn more about that.

There's a really nice implementation of phrase endpointing here: https://github.com/pipecat-ai/pipecat/blob/d378e699d23029e8ca7cea7fb675577becd5ebf… It uses three signals as input: silence interval, speech confidence, and audio level. Silence isn't literally silence -- or shouldn't be. Any "voice activity detection" library can be plugged into this code. Most people use Silero VAD. Silence is "non-speech" time. Speech confidence also can come from either the VAD or another model (like a model providing transcription, or an LLM doing native audio input). Audio level should be relative to background noise, as in this code. The VAD model should actually be pretty good at factoring out non-speech background noise, so the utility here is mostly speaker isolation. You want to trigger on speech end from the lou

Pipecat looks awesome! I'll run the examples over the weekend and try to see what the integration hooks need to look like: https://github.com/pipecat-ai/pipecat/tree/main/examples It should be pretty straightforward at first glance!

Can you please elaborate on what you would like help with? I linked all the code in the post so you can experiment with it yourself and even extend it. Also we are making use of the Pipecat framework (https://github.com/pipecat-ai/pipecat) which means you can swap in/out any STT, LLM or TTS model you would like to use:)

Here's a translation demo in Pipecat using the now ancient and arthritic GPT-4 Turbo model. :-) https://github.com/pipecat-ai/pipecat/tree/main/examples/tra... As soon as GPT-4o audio input is available through the APIs, we'll add 4o support to Pipecat. For bidirectional real-time audio, I think they'll need to make new WebSocket or WebRTC endpoints available.

I've been obsessed for the past ~year with the possibilities of talking to LLMs. I built a bunch of one-off prototypes, shared code on X, started a Meetup group in SF, and co-hosted a big hackathon. It turns out that there are a few low-level problems that everybody building conversational/real-time AI needs to solve on the way to building/shipping something that works well: low-latency media transport, echo cancellation, voice activity detection, phrase endpointing, pipelining data between models/services, handling voice interruptions, swapping out different models/services. On the theory that something like a LlamaIndex or LangChain for real-time/conversational AI would be useful, a few of us started working on a Python library for voice (and multimodal) AI assistants/agents. So ... Pipecat: a framework for building things like personal