Every AI classifier that flags, scores, or restricts real people — fraud detection, content moderation, identity verification, credit and eligibility decisions — has a permanent error rate no amount of model improvement gets to zero. The fix for the trust problem this creates isn't a better model. It's designing what happens to the person on the wrong side of the mistake. Here's the audit we run before any AI judgment feature ships.
We keep running into the same mistake across very different products, and it's worth naming directly: when an AI feature makes a call about a real person and gets it wrong, the instinct is almost always to try to improve the model. That's the wrong lever, and it's worth being specific about why — because it shows up everywhere software makes a judgment about someone: content moderation, fraud detection, identity verification, credit decisions, support automation, eligibility screens.
Accuracy was never going to save this
Every classifier has an error rate. A detector that's 95% accurate still gets one in twenty people wrong, and at any real scale, that's not a rounding error — it's a meaningful number of real people with a false label attached to something that matters to them. No amount of model improvement gets a probabilistic system to zero. If a product's trust story depends on the AI never being wrong, that story is built on a number that can't be delivered.
That's true of every judgment system we've worked on, not just content moderation. A fraud model on a payments product, a document check on an identity flow, an eligibility screen on a lending product — all of them carry a real, permanent error rate. The teams that get this right stop trying to engineer the error rate to zero and start engineering what happens on the day it's wrong.
Trust doesn't average out
Here's the part that's easy to miss if you're only looking at accuracy dashboards: users don't experience a system's trustworthiness as an average. They experience it as the worst thing that happened to them personally.
Someone who gets falsely flagged doesn't think "well, the system is right 95% of the time." They think "the system got this wrong, in a way that affects me, and I have no clear way to fix it." That one bad experience does more damage to trust than a hundred correct classifications did to build it — the same asymmetry behind why one bad experience sticks in memory longer than ten good ones.
This is why the fix for a trust problem in an AI judgment feature is almost never "improve the model." It's "change what happens to the person the model got wrong." Those are completely different design problems, and most teams only build for the first one.
What this looks like when it's done with real transparency
On Cohere Commerce, a B2B platform connecting retail buyers with brand vendors, we designed the "Cohere Score" — an AI-computed composite score, built from eight dimensions, that helps buyers evaluate vendors at a glance. The part we're proudest of isn't the score itself, it's what sits underneath it: buyers can see the dimension-level breakdown (quality, pricing, in-store experience, packaging, innovation), the pros and cons the AI surfaced, and the underlying reviews the score was built from, not just a bare number.
That's the right instinct — a score a real business can actually see the reasoning behind, rather than a number handed down with no explanation. It's also a useful example of how far "transparency" alone gets you, and where the next layer of design work usually needs to go: showing your reasoning is necessary, but it isn't the same as giving someone a clear, structured way to formally dispute a score they believe is wrong. That's the harder, less glamorous half of the problem, and it's the half most teams — including, honestly, most first versions of scoring features we've seen across the industry — under-build relative to the scoring logic itself.
We've also designed the other half directly: on Steady, an income-verification platform for gig workers, we built a review dashboard for human case workers to open a submitted income report, validate it, and either mark it complete or flag it for follow-up, with flagged items and unresolved alerts surfaced clearly so nothing sits unreviewed. It's worth being precise about what that is and isn't: it's a human reviewing a person's submission, not an AI scoring one, and the product was at an early stage when our engagement ended, before it reached real users. But the underlying pattern — a visible queue, a clear flagged state, and a human who actually looks at it — is the same pattern we'd want sitting behind any automated score that affects someone's income, credit, or standing.
The audit we run before any AI judgment feature ships
On products that make automated calls about real people, we run through the same five questions before anything ships. They're not about the model. They're about what happens after it decides.
What this actually costs to build, scaled to the stakes
Most teams under-invest here because the appeal path doesn't show up in the demo. A confidence score and a binary flag is a much shorter build than a reason explanation, a human review queue, and a resolution-time commitment. But the cost shows up eventually — as churn, as complaints, or as a regulator's attention, instead of as a line item in a sprint plan.
We think about the right investment level based on what the flag actually costs someone:
Low-stakes flags — a spam filter, a low-confidence content warning — can reasonably ship with a lightweight appeal: a button, a form, a promise of review within a few days. The cost of being wrong is annoyance, not damage.
Medium-stakes flags — a fraud hold on a transaction, an account restriction — need a real human review path with a defined resolution time, because the visible cost to the wrongly-flagged person is now measured in real inconvenience or real money.
High-stakes flags — anything touching identity, eligibility, or someone's ability to earn a living — need a human in the loop before the consequence lands, not after. If reversing the decision after the fact can't undo the damage, the review has to happen before the flag takes effect, not once someone complains.
The bigger pattern
We keep seeing the same root cause across very different products: the team building the AI feature spent its design time on the decision and almost none on the consequence. It's an easy trap, because the decision is the interesting engineering problem, and the consequence feels like an afterthought — error handling, not "real" product work.
It isn't an afterthought. On any product where software makes a call about a person, the consequence design is the product, at least as much as the decision logic is. Users don't experience your model. They experience what happens to them when it's wrong.
If your team is shipping, or already runs, a feature that flags, scores, or restricts people automatically, the five questions above are a reasonable starting audit — steal them as-is. If working through what a real appeal path should look like for your specific product is the part you're stuck on, that's the kind of problem we spend most of our time on with clients building data-heavy, judgment-heavy products.
Gytis Markevičius
Founder, GytisMark Studio — gytismark.com
Other articles
Every engagement includes a senior designer thinking about the product strategically, catching issues before they reach you, and making sure the work holds up under scrutiny.
Let's talk about your product and your business goals.
Request your FREE trial →✦ Start with a conversation ✦
20 minutes. No pitch. We'll tell you honestly if we're the right fit.
