examples

g-a-real-time-shopping-... Code here: https://github.com/CerebriumAI/examples/tree/master/27-ecomm... Basic Flow We use a variety of tools in order to achieve the end-to-end functionality. - *Daily* - We use Daily to extract video frames from the live stream every 150ms. This allows us to run object detection and our search function on each frame. - *Yolov8*: We use a custom trained Yolov8 model to detect objects in our live stream that we must search for. - *Turso*: We store our product catalog in Turso's embedding database that searches for matches against the detected objects. - *Supabase*: We use Supabase to store detected images and we store items found via our search function in a database that allows our client application to update with the results in real-time. - *Serp API* (Bonus): We use the SERP API to search Google for a product match using only the image of the detected item. - *Cerebrium*: The entire solution is hosted on Cerebrium serverless CPU/GPUs. It manages all auto-scaling and orchestration. ### Things to improve - Instead of only detecting a class image once and never again, I would store the embeddings of previously searched items and if an image makes it through the search criteria you can compare it to past embeddings. This means you could search multiple books, cups etc. - Train a better model. We trained our model on 6 classes (book, shoe, watch, cup etc) and about 40 images. Depending on your use case, I would train it on a lot more data for your use case to get more accurate. - Facebook recently released their SAM 2 model that can follow items through a video. It would be very cool to incorporate this, showing a user what items have already been searched as time progresses. - If a user has a bad camera, the accuracy of the YOLO model is much lower. I would recommend training on a diverse set of data across different quality cameras.

7-ecomm...) Basic Flow: We use a variety of tools in order to achieve the end-to-end functionality. - Daily - We use Daily to extract video frames from the live stream every 150ms. This allows us to run object detection and our search function on each frame. - Yolov8: We use a custom trained Yolov8 model to detect objects in our live stream that we must search for. - Turso: We store our product catalog in Turso's embedding database that searches for matches against the detected objects. - Supabase: We use Supabase to store detected images and we store items found via our search function in a database that allows our client application to update with the results in real-time. - Serp API (Bonus): We use the SERP API to search Google for a product match using only the image of the detected item. - Cerebrium: The entire solution is hosted on Cerebrium serverless CPU/GPUs. It manages all auto-scaling and orchestration. Things to improve - Instead of only detecting a class image once and never again, I would store the embeddings of previously searched items and if an image makes it through the search criteria you can compare it to past embeddings. This means you could search multiple books, cups etc. - Train a better model. We trained our model on 6 classes (book, shoe, watch, cup etc) and about 40 images. Depending on your use case, I would train it on a lot more data for your use case to get more accurate. - Facebook recently released their SAM 2 model that can follow items through a video. It would be very cool to incorporate this, showing a user what items have already been searched as time progresses. - If a user has a bad camera, the accuracy of the YOLO model is much lower. I would recommend training on a diverse set of data across different quality cameras.

flexible than existing solutions. Vapi, Retell, [Bland.ai](http://Bland.ai) are too opinionated plus since they just orchestrate API’s which incur network latency at every step. See requirement above 3. The unit economics actually work at scale. So we decided to create a implementation of our own. Process: As we mentioned in our previous release, if you want to achieve response times this low you need to make everything as local as possible. So below was our setup - Local STT: Deepgram model - Local Embedding model: Nomic v1.5 - Local VectorDB: Turso - Local LLM: Llama 3B - Local TTS: Deepgram model From our previous example, the only new components where: - Local Embedding model: We chose Nomic Embed text v1.5 model that gave a processing time of roughly ~200ms - Turso offers local embedded replicas combined with edgeDB’s which meant we were able to achieve 0.01 second read times. Pinecone also gave us good times of 0.043 seconds. The above changes led us to achieve sub 1 second voice-to-voice response times Application: With Andrej Karpathy’s announcement around [Eureka Labs](https://eurekalabs.ai/), a new AI+Education company we thought we would create our very own personalised Andrej tutor. Listen to anyone of his Youtube lectures, as soon as your start specking, the video will pause and he will reply. Once your question has been answered you can then tell him to continue with the lecture and the video will automatically start playing. Demo: https://educationbot.cerebrium.ai/ Blog: https://www.cerebrium.ai/blog/creating-a-realtime-rag-voice-... Github Repo: https://github.com/CerebriumAI/examples/tree/master/19-voice... For demo purposes: - We used OpenAI for GPT-4-mini and embeddings (its cheaper to run on a CPU than GPU’s when running demos at scale. These changes add about ~1 second to the response time - We used Eleven labs to clone his voice to make replies sound more realistic. This adds about 300ms to the response time. The improvements that can be made which we would like the community to contribute to are: - Embed the video screens as well that when you ask certain questions it can show you the relevant lecture slide for the same chuck that it got context from to answer. - Insert the timestamps in the vectorDB timestamps so that if a question will be answered later in the lecture he can let you know This unlocks so many use cases in education, employee training, sales etc that it would be great to see what the community builds!