Cheap to Build, Hard to Change: Keeping AI-Built Systems Legible

When building gets cheap, nobody is forced to plan for durability, and the result is systems that work today and can't be explained next quarter. Here's how I think about designing AI features so the people who depend on them can still see what's going on.

‍

Nielsen Norman Group published a piece in early October on why agentic AI systems turn fragile, and one observation in it has stayed with me: because AI removes the effort from building, people overbuild, and it feels productive while they do it. The authors link this to an old software idea, the big ball of mud, where a system grows by accretion until nobody understands it well enough to change it safely.

I don't think this is only a problem for the people building the systems. It's a product design problem, because the people who suffer from a system nobody can explain are its users.

How a system turns into mud

The pattern the article describes is familiar to anyone who has inherited a spreadsheet that runs part of a business. Something gets put together for an immediate need. It works, so nobody wants to touch it. New requirements get attached to the old foundation, patches keep it running, a folder fills up with things only the tool itself can find, and eventually the whole thing is rebuilt or abandoned.

What's different now is how little it takes to start. A person with no technical background can assemble a multi-step agentic workflow in an afternoon. The constraint that used to force a little planning, that building was slow and expensive, has gone.

The question the article offers as a test is a good one: if you wanted to change how this system works, would you know what to change? I'd extend it. Would the person using it know what it's doing right now, and why?

Where design comes in

A fragile system usually has a fragile interface, and the two fail together. These are the places I look when I'm reviewing an AI feature.

Can someone see what the system is allowed to do on its own? If an agent can send, change, or delete things, the permissions should be visible in the product and not buried in a prompt only the builder has read.

Can someone see where its instructions come from? When behaviour depends on a mix of global settings, per-project rules, and context picked up along the way, users need a way to tell which one is driving a given result. The article makes a similar point about keeping those kinds of context separate and auditing them periodically.

Can someone change one thing without breaking three others? If the honest answer is that nobody knows, the product has already accumulated the debt, even if it looks tidy on the surface.

What happens when it's wrong? Mud tends to show up first in failure. A system that can't explain a bad result to its user is a system its own team can't debug either.

A habit worth building into the process

It doesn't take a heavy process to stay ahead of this. One habit I'd suggest is a short, regular review where someone asks to be shown the current structure of the system as a newcomer would see it. Not the output, but the instructions, permissions, and sources behind it. Anything that can't be explained in a couple of sentences is a candidate for simplifying before more gets stacked on top.

The second is to treat the first working version as a prototype. That sounds obvious, but the whole appeal of cheap building is that the prototype immediately starts doing real work, and then replacing it feels like a step backwards. Deciding in advance what would trigger a rebuild makes it a plan instead of an emergency.

Cheap building is a real gain, and none of this is an argument against it. It just moves the cost to a later date and a less convenient moment. If someone new joined your team tomorrow, could they tell, from the product alone, what your AI features are allowed to change and where they get their instructions?

‍

Gytis Markevičius
Founder, GytisMark Studio — gytismark.com