UFO

Here you go: UFO - A UI-Focused Agent for Windows OS Interaction "UFO is a UI-Focused dual-agent framework to fulfill user requests on Windows OS by seamlessly navigating and operating within individual or spanning multiple applications." https://github.com/microsoft/UFO?tab=readme-ov-file

Right, but that's not an OS, that's a software application run by the OS like Microsoft's UFO[0]. The ability to "teach" the LLM by converting your teachings to those skills is very clever though. 0. https://github.com/microsoft/ufo

I think this is a great primer on how to make LLMs enterprise/domain specific, but is still couching this technology's main use-case in the context of "question-answer" machine toward the service of a user. While this may be the (brilliant) hook that enabled the massive conceptual adoption of the tech it is not going to be the value proposition of it. In the next 6 months we're going to see the deployment of Agency-based solutions completely powered by LLM[1][2]. The real value LLM provide will be that they replace the "humans-as-integrations" and RPA automations. 1. https://github.com/microsoft/UFO 2. https://github.com/microsoft/TaskWeaver

The code isn’t very complicated. You could edit in any vision model you want: https://github.com/microsoft/UFO/blob/main/ufo/llm/llm_call....