Voyager

Some "agents" like the minecraft bot Voyager(https://github.com/MineDojo/Voyager) have a control loop, they are given a high level task and then they use LLM to decide what actions to take, then evaluate the result and iterate. In some LLM frameworks, a chain/pipeline just uses LLM to process input data(classification, named entitiy extraction, summary, etc).

In a way it is the same thing, agents are mostly an abstraction that make it easier to know what’s going on. I think of agents more or less as python classes with a mixture of natural language and code functions. You design them to do something with information they produce, and to interface with other agents or “tools” in some way. But all the agents can be the same language model under the hood, they are frames used to build different kinds of contexts. And yes I think the idea is that emergent behaviour can be useful. This comes to mind https://github.com/MineDojo/Voyager But I think we are still a small ways off from being really smart about agents. My opinion is that we haven’t quite figured out what we are doing yet.

I read through the code and tried it out for 15 mins. It's a hard-coded program that can do a text search for it's own hard-coded, human-implemented functions. Apparently it can string those functions together, but doesn't do it correctly. https://github.com/MineDojo/Voyager/tree/main/voyager/contro... 20 minutes of light reading through the repository pretty much dispels any notions that this is a self-learning system that can reason and think. It's the same minecraft automation we have been seeing for a decade now, with a chatbot text search builtin.

Even more than the complex prompts, they give the bot access to a variety of functions implementing primitive actions, which are crafted to give helpful hints if the bot is struggling. For example, the code for mining a block: https://github.com/MineDojo/Voyager/blob/main/voyager/contro... This function keeps a global count of how many times it's been called to mine a block that doesn't exist nearby, and will warn the bot that it should try exploring first instead. There's nothing necessarily wrong with all that - it's an important research question to understand how much hand-holding the agent needs to be able to do these sorts of tasks. But readers should be aware that it's hardly dropping an agent into the world and leaving it to its own devices.