M. Semih Babacan

Why I treat agent orchestration as a distributed systems problem

RuFlo bets that coordinating many agents is a systems problem, not a prompt problem. I think the bet is right, and I also think most teams reach for it two months early.

Mehmet Semih Babacan
Mehmet Semih Babacan

AI Technical Product Manager

March 8, 20268 min read21 views
Why I treat agent orchestration as a distributed systems problem

Almost every team I talk to has one agent that works. One prompt, one tool loop, one job it does well enough that somebody demoed it and everybody clapped. Then they try to make five of those work together and the whole thing turns to mush. The usual fix is more prompt: a coordinator agent, a longer system message, a paragraph that starts with "you are the orchestrator". It holds for about a week.

So, my position up front. Coordinating many agents is a state problem wearing a prompt costume. Once you accept that, you stop writing coordinator prompts and start writing the boring things distributed systems people have been writing for forty years. RuFlo, the project formerly called Claude Flow, is the clearest bet on that reading I have seen packaged as a product, and this post is me arguing with it in public.

What the wall actually feels like

The failure is never dramatic. Nobody's agent catches fire. What happens is that the tester agent checks code the coder agent already replaced, the reviewer approves a diff that no longer exists on disk, and the architect plans around a file that was deleted forty seconds ago. Every individual agent behaved correctly given what it could see. That is the tell. When each part is right and the whole is wrong, you are not looking at a reasoning failure, you are looking at a consistency failure.

Prompting cannot fix a consistency failure, because the prompt is not where the disagreement lives. The disagreement lives in the fact that five processes read five slightly different snapshots of the world and none of them knows it. You can make each agent smarter forever and the mush will not clear.

What RuFlo is selling

I want to be careful here, because I have read the project's own description of its architecture and I have not benchmarked it. What follows is the pitch, marked as the pitch.

RuFlo describes a fleet of specialised agents rather than one general one: a coder, a tester, a reviewer, an architect, a security role, and dozens more beyond those. They are not supposed to coordinate by talking to each other in prose. They coordinate through consensus and replication primitives with names you would recognise from a database paper: Raft, byzantine fault tolerance, gossip, CRDTs. On top of that sit the things that make a procurement team relax, which is to say a security layer, audit trails, and a claims mechanism for passing work between a human and an agent.

The interesting part is not the agent count. A catalogue of roles is cheap, and I would not choose a platform on it. The interesting part is the choice of vocabulary. Picking Raft over "the coordinator decides" is a statement that the hard problem is agreeing on what happened, not deciding what to do next.

Agreeing on what happened is harder than deciding what to do next.

my one-line summary of the bet

Where the coordination cost actually lives

Here is the shape of it. If your agents talk to each other, every new agent has to be introduced to every existing agent. Four agents means six conversations. Ten agents means forty-five. Nobody sits down and designs that, it just grows, one helpful handoff at a time, until adding an eleventh agent means touching every prompt you own.

Route the same agents through one shared, versioned store and the count goes linear. Ten agents means ten connections, and the eleventh costs exactly one.

Left: every agent knows every other agent, so links grow as n(n-1)/2. Right: every agent knows one store, so links grow as n. That second shape is the whole argument for an orchestration layer.

The store is not a message bus, and that distinction is where I watch home-grown attempts go wrong. A bus moves messages and forgets them. What you need is something that remembers, that can tell you the current version of a fact, and that can refuse a write when the writer was working from a stale read. That last clause is the whole product. An agent holding a stale read is exactly the reviewer approving a diff that no longer exists.

The handoff is the whole problem

Look at the two ways a piece of work moves from one agent to the next. In the version most teams ship first, the handoff is a paragraph of prose stuffed into the next agent's context.

prose handoff
coordinator.prompt = `
You are orchestrating a team.
The coder just finished the auth
refactor. Ask the tester to check
it, then have the reviewer look
at whatever the tester says.
Use your judgement.
`;
state handoff
task = {
id: "auth-refactor-7",
produced_by: "coder",
base_version: 41,
artifacts: ["src/auth/*.ts"],
claim: { role: "tester",
         expires_at: 1741_446_000 },
accept_if: "tests_pass",
};
The same handoff, twice. The right-hand version can be validated, replayed and refused. The left-hand one can only be re-read.

The left-hand version carries no version number, so nothing can notice that the tester is working against revision 41 while the coder has already moved to 43. It carries no claim, so two agents can pick up the same task and neither will find out. It carries no acceptance condition, so "done" is decided by whichever model happens to answer. And when it fails at 3am you get a transcript, not a trace.

The right-hand version is duller to read and that is the point. base_version makes stale work detectable. The claim makes duplicate work impossible instead of unlikely. accept_if moves the definition of done out of a model's opinion and into the system. None of this is clever. All of it is the difference between a demo and something you would leave running overnight.

When orchestration earns its complexity

Okay, the honest part, because everything above reads like an advertisement and I do not want it to. An orchestration platform is a large piece of machinery, and plenty of teams adopt one while their real problem is still that a single agent has a bad prompt. Machinery does not fix that. It hides it, then bills you for the hiding.

The test I use has four questions, and I want all four to come back yes before anybody installs anything.

  1. Do two agents write to the same thing?

    Not read. Write. If your agents only read shared context and write to separate places, you have a fan-out job, and a fan-out job wants a queue, not a consensus protocol. The moment two roles can touch the same file, the same row, the same branch, you have a conflict problem, and conflicts want machinery.

  2. Does the work outlive one session?

    A run that finishes inside one context window can keep its state in that context window. When work spans hours, restarts, or a human going home, the state has to live somewhere that survives the process. That somewhere is an orchestration layer, and if you refuse to adopt one you will write a worse version of it yourself.

  3. Does anyone have to answer for what happened?

    In a side project, "the agent did something odd" is a shrug. In a regulated one it is an incident with a deadline. Audit trails, claims and permission boundaries stay optional right up until the first time somebody asks who approved a change, and after that they are the entire conversation.

  4. Is the failure mode expensive?

    If a bad run costs a retry, prompt it and move on. If a bad run costs money, data, or a customer, you are buying insurance, and the complexity is the premium you pay for it.

Four yeses and orchestration is the cheapest thing you can do. Three yeses and I would wait, because the fourth is usually about to arrive and you will feel it when it does. One or two, honest answer: you are decorating.

What I cannot tell you

I have read RuFlo's description of its own architecture and I find the reasoning behind it convincing. I have not run it at scale, I have not measured its overhead, and I have not watched what a swarm does when the model underneath it has a bad day. Take the systems argument in this post as mine, and take the specific product claims as the project's, because that is what they are.

The part I am confident about is the direction. If models keep converging, and I think they will, what separates a team that ships agents from a team that demos them is not model choice. It is whether they can say which agent did what, against which version, and who accepted it. That was never a prompt.

The open question I have no answer for is build or buy. Buying gets you primitives you would take a year to write badly. Building keeps the coordination layer, which is the part most likely to become your moat, inside your own repo. I lean buy for the first year and build the pieces that hurt, but ask me again after somebody's swarm has failed in front of me.