open-operator-evals

We’ve open-sourced a benchmark for LLM-driven web agent setups. It evaluates real-world tasks, like logging in, scraping dashboards, and submitting forms, using structured criteria: success rate, latency, and task reliability. Everything is fully reproducible, with all outputs, logs, and evaluation data available. https://github.com/nottelabs/open-operator-evals Feedback, critiques, or contributions welcome:)

Hey HN! I built Notte to see if converting DOM into natural language could improve web agent capabilities and make them work reliably with smaller models. The result was using deep DOM parsing and a semantic abstraction layer to transform websites into structured, navigable maps described in NL. Instead of feeding raw HTML, there is a perception layer that means LLMs don't just click the DOM elements, but understand the intent behind them. I benchmarked it against other agent frameworks and was pleasantly surprised - faster task completion and increased reliability (all open source with replayable/reproducible code). Beyond the core tech, I also built out unified session management, stealth features, credentials vault, CAPTCHA HITL + some more cool features all via a single API. Still working out some edge cases with dynamic content, but