ADaddcn·Oct 25, 2023
Hey HN - I’ve been working on adding support for natural language rules to Optic’s API linting + testing tool [0].
Instead of generating API designs, we’re using LLMs to check if a team’s API is “good” by their own standards.
Many large companies that use OpenAPI have custom Spectral + Optic rules to enforce API security practices, consistent styles, versioning policies, SLA adherence, and to prevent breaking change. These tests are hard to write and cannot verify a lot of the more abstract aspects of an API design. It’s really hard to write code to check if an OpenAPI operation allows batch creation, but when you give an LLM YAML and the rule “No endpoint should allow users to create multiple resources at once” — it does a surprisingly good job at testing the design and writing a nice error message when it fails.
The hardest technical c
Read the rest
hallenge has been:
- figuring out how to fit OpenAPI files (sometimes 10k lines or more) into the context window. Optic already breaks up API specs into their parts: ie parameters, response bodies, headers, etc. We’ve been putting just the relevant lines of the OpenAPI file into the LLMs and resolving all their dependencies beforehand so the model has all the context it needs.
- Minimizing API calls. The first time you run LintGPT it is pretty slow because it has to run every rule across every part of the API specification (1000s of calls). But we shouldn’t have to repeat that work. Most of the time parameters, properties, etc don’t change and neither do the rules. We’re building caching into our web app to make this fast / save $ for end users.
Happy to answer any questions. I really think there’s a huge use case here for linting all kinds of code, config, database schemas, policies in ways that were never possible before. And personally, I like the idea of having these smart tools guiding me towards making my work better vs generating it all for me — idk something about that just feels good.
[0] https://github.com/opticdev/optic