screenpipe

(https://forum.obsidian.md/t/fine-tuning-openai-api-gpt3-on-y...). Later I built Ava, the first Obsidian AI plugin, which grew to a few thousands of users quickly. It then became Embedbase, an API to make it easier to build AI apps powered by RAG. What I learned from all this is how important it is for the models to have context about what you’re doing on your computer, in order to get them to do what you want. In the early days there was fine tuning but it was too much pain, then there was tool calling so that AI can access software you use but still kinda not autonomous enough. needing micro management. Then MCP came, but it felt too static, and non technical users struggled to build and use MCP. Then we got skills. Most recently we’ve seen Karpathy’s LLM-maintained wiki, Garry's GBrain, etc., where an agent incrementally maintains a persistent collection of Markdown pages. New sources update entity pages, strengthen or contradict existing claims, and improve a synthesis that compounds over time. I like this pattern, but it still begins with someone selecting and importing the sources. There is still no way AI can know what you and your company are doing every day, across apps, not just inside of apps. Of course, not everyone wants this. But I do! I want AI to know what I'm doing and never lose memory ever again, and I want it to use the same software that humans do, without painful context switches. I started building Screenpipe for myself in 2024 - a CLI to record your screen and plug this context into AI. An HN user posted it in 2024 (https://news.ycombinator.com/item?id=41695840) and that discussion influenced the product. The most useful criticism concerned recording consent, local security, CPU usage, signal-to-noise, and whether agents could act on top of the data. The naive implementation started from continuously recording video and running OCR over every frame. But that creates duplicate data, consumes substantial resources (it basically turns your computer into a space heater!), and discards structure the operating system already knows. Screenpipe now instead listens for events such as app switches, clicks, typing pauses, scrolling, and idle fallbacks. When something meaningful changes, it pairs a screenshot with the operating system’s accessibility tree at the same timestamp. OCR is used when structured accessibility data is unavailable. We also capture audio continuously, identify speakers and transcribe locally through Parakeet/Whisper or using cloud models. Everything is indexed in a local SQLite database, mp4 files, and sometimes md files. An AI friendly API on port 3030 is open for agents, with authentication and a MCP and skills. Once Screenpipe has been up and running for a while, you can use it through our built-in chat, Claude, ChatGPT, Hermes, Openclaw, or any agent, to do things like: - adding context to your current chat, e.g. "gather all context about task X", then requiring less prompts to achieve your goal - retrieve information, e.g. "retrieve the tasks i was working on from 8 am to 4 pm, make a list of what got done and what's left" - create and maintain a personal wiki / second brain for your agents: "every 1h organize everything i do in projects, people, tasks, meetings in my Obsidian vault as markdown files and folders" - create automations: whenever i visit someone's profile on linkedin, update my crm - find automation opportunities: look at everything my team has done this week and turn it into a list of automation opportunities Screenpipe data is stored locally, though we also offer an enterprise plan to discover automation opportunities and for that the company decides where the data lives. We built our own AI PII model to redact sensitive information, it runs locally on Apple MLX or Windows DirectML, we also support cloud confidential inference for low end devices, although our local models are meant to use <1% CPU and <400 mb RAM. Users can set apps, windows, and urls to filter, in addition to browser incognito mode. We also support recording schedules and other privacy features. Most of our codebase is written in Rust, MLX, Onnx, we like cidre or direct C call for Apple APIs and windows-rs for Windows API. We also experimentally support Linux. We have a desktop app (https://screenpipe.com/how-to-install) and a CLI: npx screenpipe record You can run that without creating an account. All the code is source-available at https://github.com/screenpipe/screenpipe. We took the dreaded step of making our own Screenpipe Commercial License. I know HN strongly prefers OSI open source (MIT/Apache/etc.) but couldn’t find a sustainable way to keep developing Screenpipe while companies were using it commercially for free. So now personal non-commercial, nonprofit, educational, and research use is free, but commercial use requires a license. Versions released before the license change remain available under MIT. We have a free tier, and other plans, including Enterprise which helps companies find automation opportunities. Would love to hear any feedback, things you've done with screenpipe, or features you'd want

OCR to your app (likely free) as a business (aight) (Note that there's also bounties, which I thought was pretty cool.) So apparently it's... USD 205?! The price is going up significantly too, up to USD 332, intended to induce FOMO. (This also leaves a bad taste in my mouth.) Apparently, you could also get it free, albeit without priority support, if you build it yourself. But it's MIT-licensed, and thus supposedly libre, right? So there's nothing stopping a trusted community member, or package managers such as Chocolatey, from just redistributing the binary. This used to be how Aesprite worked, before they changed to a non-libre license to prevent people from doing that. Luckily, it seems like Screenpipe has no CLA, so changing licenses in this way doesn't seem likely. So in general... this is all very confusing, but from what I've understood, there is no reason why people can't just redistribute the binaries for free, it's just that someone hasn't done it yet. So the USD 332 just gets you 1 year of priority support. (Which also means that, yes, don't buy it now during the "sale". It'll drop to USD 0 soon, if you just care about getting the binaries.) Sorry for the inflammatory text, I just find these business practices really shady, especially for a project that advertises itself using open source. The confusion and "FOMO sale" [2] is mildly bad, but the "get our app by posting us on Reddit ten times" part is awful. [1] https://github.com/mediar-ai/screenpipe [2] https://web.archive.org/web/20241001013505/https://screenpi.pe/onboarding