HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
langwatch
scenario
TypeScript2025–2026
langwatch / scenario

scenario

Agentic testing for agentic codebases

Web DevelopmentAI & Machine LearningScience & EngineeringTesting & QualityLibraries & SDKs
Explore on GitHubRead Hacker News discussions
4Distinct people
9Mentions
0Explicit recommenders
1KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Agentic testing for agentic codebases

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month9 dated mentions

Observed mentions: 2025-04-15 13:26:43 UTC to 2026-07-22 14:52:14 UTC.

012342025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsApr 20252025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 4 mentions; 1 submissions, 3 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsNov 20252025-12-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-01-01 00:00:00 UTC to 2026-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-02-01 00:00:00 UTC to 2026-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-03-01 00:00:00 UTC to 2026-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-04-01 00:00:00 UTC to 2026-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-05-01 00:00:00 UTC to 2026-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-06-01 00:00:00 UTC to 2026-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-07-01 00:00:00 UTC to 2026-08-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsJul 2026

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC110
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC000
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC413
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC303
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC000
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC000
Dec 20252025-12-01 00:00:00 UTC2026-01-01 00:00:00 UTC000
Jan 20262026-01-01 00:00:00 UTC2026-02-01 00:00:00 UTC000
Feb 20262026-02-01 00:00:00 UTC2026-03-01 00:00:00 UTC000
Mar 20262026-03-01 00:00:00 UTC2026-04-01 00:00:00 UTC000
Apr 20262026-04-01 00:00:00 UTC2026-05-01 00:00:00 UTC000
May 20262026-05-01 00:00:00 UTC2026-06-01 00:00:00 UTC000
Jun 20262026-06-01 00:00:00 UTC2026-07-01 00:00:00 UTC000
Jul 20262026-07-01 00:00:00 UTC2026-08-01 00:00:00 UTC110

What Hacker News says

9 sources
jangletown·Jul 22, 2026Submission

Show HN: Langy, an automated AI engineer (we gave it a robot body) [video]

Founder here. Langy is an AI engineer that lives inside our platform, LangWatch. It reads your production traces, writes Scenario tests and evaluations for the problems it finds, opens a pull request on your repo, and proves the fix by running those simulations in CI. A human still merges. Reachy robot from Hugging Face arrived on the same week we were planing on launching it, so we wanted to use for the launch. So we it wired to our Langy which actually lives on the platform, and asked it to test our own customer-support voice agent. On the video you can see it writing the agent tests reading traces and everything. LangWatch is an open source platform (github.com/langwatch/langwatch) and Scenario, the simulation-testing library it drives, is open source too if you want to run that part yourself: https://github.com/langwatch/scenario. A f
Read the rest
ull write-up on how Langy works under the hood is coming later this week. Happy to get into how it drives the robot, what is real vs staged, latency, or how the whole harness around it actually works.
9 points · Open discussion
draismaa·Jul 28, 2025
We open-sourced Scenario, a tiny framework to simulate and test AI agents by using another AI agent — much like how self-driving cars are tested in controlled environments before real-world deployment. The idea: if you wouldn’t deploy an autonomous vehicle without simulation, why would you deploy an AI agent without pressure-testing it first? Scenario lets you: Describe multi-turn user flows (e.g. “book a flight and cancel it”) Set success criteria ("user got confirmation + refund info") Have one agent simulate the user, testing how another agent performs the task Debug regressions, track failures, iterate on behavior It’s like writing unit tests, but for conversations. Why? Traditional evals fall short when testing multi-step flows, memory handling, tool use, or goal alignment. Inspired by simulation in autonomous vehicle testing, where
Read the rest
cars are exposed to rare, edge-case situations at scale, Scenario allows similar validation loops for AI agents. Instead of hoping your support bot or agent workflow handles complexity… you simulate it: Works with any agent-framework you define Minimal setup required. There are a few starter examples in the repo: https://github.com/langwatch/scenario We’d love feedback from the HN community: What test patterns do you (want to) use for agent workflows? Thanks!
In [dead]
jangletown·Jul 9, 2025
hello aszen, I work with draismaa, the way we have developed our simulations is by putting a few agents in a loop to simulate the conversation: - the agent under test - a user simulator agent, sending messages as a user would - a judge agent, overlooking and stopping the simulation with a verdict when achieved it then takes a description of the simulation scenario, and a list of criteria for the judge to eval, and that's enough to run the simulation this is allowing us to tdd our way into building those agents, like, before adding something to the prompt, we can add a scenario/criteria first, see it fail, then fix the prompt, and see it playing out nicely (or having to vibe a bit further) until the test is green we put this together in a framework called Scenario: https://github.com/langwatch/scenario the cool thing is that we also built
Read the rest
in a way to control the simulation, so you can go as flexible as possible (just let it play out on autopilot), or define what the user said, mock what agent replied and so on to carry on a situation and then in the middle of this turns we can throw in any additional evaluation, for example checking if a tool was called, it's really just a simple pytest/vitest assertion, it's a function callback so any other eval can also be called
In Agent simulations = unit testing for AI?
0xdeafcafe·Jul 3, 2025
Totally agree. We've found that a lot of "agent failures" trace back to assumptions, bad agent-decisions, or bloat buried in the context, stuff that makes perfect sense to the dev who built it when following the happy path, but can so easily fall apart in real-world scenarios. We've been working on a way to test this more systematically by simulating full conversations with agents and surfacing the exact point where things go off the rails. Kind of like unit tests, but for context, behavior, and other ai jank. Full disclosure, I work at the company building this, but the core library is open source, free to use, etc. https://github.com/langwatch/scenario
In The new skill in AI is not prompting, it's context engineering
jangletown·Jun 26, 2025
That's true, we have been trying to help customers doing evals for ages now, and it's super hard for everyone to build a really good dataset and define great quality metrics just wanted then to shameless plug this lib I've built recently for this very topic, because it's been much easier to sell that into our clients than evals really, because it's closer to e2e tests: https://github.com/langwatch/scenario instead of 100 examples, it's easier for people to think on just the anecdotal example where the problem happens and let AI expand it, or replicate a situation from prod and describe the criteria in simple terms or code
In Karpathy: "context engineering" over "prompt engineering"
rchaves·Jun 24, 2025
Hello HN! tl;dr: We built Scenario, an open-source testing library for AI agents. It simulates real conversations with your agent, its code-driven, and lets you assert anything mid-dialogue. Repo: https://github.com/langwatch/scenario Docs: https://scenario.langwatch.ai/ I'm Rogerio, founder of LangWatch, I've been helping many customers building LLM applications in this past two years and worked with Alex on this. Most of the efforts for LLM quality so far were about evaluations, single-turn, there was nothing actually good to test agents, it all felt forced, but we believe we cracked it now, we have built an agent testing library that test your agent by simulating a user and playing a conversation back and forth with it. One of the key challenges there was that we had to make it compatible with all the 273+ AI frameworks (and counting)
Read the rest
there are. Luckliy AG-UI protocol popped up recently, standardizing agents frameworks and UI interactions, this is perfect, because at the end of the day, we want our user simulator to "see" just the same that the user sees. So we made Scenario in a way that is really easy to connect to any agent no matter the tech stack, from a simple string <-> string connection, to openai standard messages format, to AG-UI. The other key challenge was to balance testing the open-endedness of agents vs having reliable cases you want to test, so we worked a lot on thinking through the autopilot simulation vs the fully scripted one, and here again, the goal was complete interoperability. At the end of the day, the design we achieved was simply having lambdas, that you can call at any point of the test, so it's just code, where you can connect any other evaluation or assertion tool you want, we are not restrictive. Check out the repo and the docs, we would love to get some feedback in here! Repo: https://github.com/langwatch/scenario Docs: https://scenario.langwatch.ai/
In Show HN: Simulation-Based Testing for Agents Using AG-UI Protocol

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.