inference

One of the most common uses for edge AI not listed in this course is computer vision. You similarly want real-time inference for processing video. Another open source project that makes it easy to use SOTA vision models on the edge is inference: https://github.com/roboflow/inference

We’ve got an open source pipeline as part of inference[1] that handles the nuances (multithreading, batching, syncing, reconnecting) of running multiple real time streams (pass in an array of RTSP urls) for CV models like YOLO: https://blog.roboflow.com/vision-models-multiple-streams/ [1] https://github.com/roboflow/inference

Tried it a few weeks ago for a task (had a few dozen files in an open source repo I wanted to write tests for in a similar way to each other). I gave it one example and then asked it to do the work for the other files. It was able to do about half the files correctly. But it ended up taking an hour, costing >$50 in OpenAI credits, and took me longer to debug, fix, and verify the work than it would have to do the work manually. My take: good glimpse of the future after a few more Moore’s Law doublings and model improvement cycles make it 10x better, 10x faster, and 10x cheaper. But probably not yet worth trying to use for real work vs playing with it for curiosity, learning, and understanding. Edit: writing the tests in this PR given the code + one test as an example was the task: https://github.com/roboflow/inference/pull/533 This commit

FWIW you can use roboflow models on-device as well. detect.roboflow.com is just a hosted version of our inference server (if you run the docker somewhere you can swap out that URL for localhost or wherever your self-hosted one is running). Behind the scenes it’s an http interface for our inference[1] Python package which you can run natively if your app is in Python as well. Pi inference is pretty slow (probably ~1 fps without an accelerator). Usually folks are using CUDA acceleration with a Jetson for these types of projects if they want to run faster locally. Some benefits are that there are over 100k pre-trained models others have already published to Roboflow Universe[2] you can start from, supports many of the latest SOTA models (with an extensive library[3] of custom training notebooks), tight integration with the dataset/annotati

Yeah, inference[1] is our open source package for running locally (either directly in Python or via a Docker container). It works with all the models on Universe, models you train yourself (assuming we support the architecture; we have a bunch of notebooks available[2]), or train in our platform, plus several more general foundation models[3] (for things like embeddings, zero-shot detection, question answering, OCR, etc). We also have a hosted API[4] you can hit for most models we support (except some of the large vision models that are really GPU-heavy) if you prefer. [1] https://github.com/roboflow/inference [2] https://github.com/roboflow/notebooks [3] https://inference.roboflow.com/foundation/about/ [4] https://docs.roboflow.com/deploy/hosted-api

Hi! Supervision does not run models, but it connects to existing detection and segmentation libraries, allowing you to do more advanced stuff easily. Take a look here to get a high-level overview: https://supervision.roboflow.com/latest/how_to/detect_and_an.... As for Roboflow, you can use the `inference` package to run (among other things) all Roboflow Universe models locally. Take a look at README examples: https://github.com/roboflow/inference.

This looks like a really cool project; would you be open to us PR'ing support for the 50k fine-tuned models on Roboflow Universe[1] via an `inference`[2] integration? [1] https://roboflow.com/universe [2] https://github.com/roboflow/inference

Deploying vision models is time consuming and tedious. Setting up dependencies. Fixing conflicts. Configuring TRT acceleration. Flashing (and re-flashing) NVIDIA Jetsons. A streamlined, developer-friendly solution for inference is needed. We, the Roboflow team, have been hard at work open sourcing Inference, an open source vision deployment solution. Our solution is designed with developers in mind, offering a HTTP-based interface. Run models on your hardware without having to write architecture-specific inference code. Here's a demo showing how to go from a model to GPU inference on a video of a football game in ~10 minutes: https://www.youtube.com/watch?v=at-yuwIMiN4 Inference powers millions of daily API calls for global sports broadcasts, one of the world’s largest railways, a leading electric car manufacturer, and multiple other Fort