Banned by the Algorithm: Why "Appeal Later" Isn't Trustworthy AI

"I was banned from Vinted over a £2 pair of sandals"

If you missed that headline on BBC News(this will open in a new window) this week, it's worth a read. A loyal Vinted seller, with a good track record, gets permanently banned over a low-value item. No warning, no real explanation, and a decision that appears to have been made by a system rather than a person.

Vinted has over 17 million users in the UK, and most have no problems at all. But a growing number of experienced, well-meaning sellers are reporting the same pattern: wrongly suspended or permanently banned accounts, funds frozen, and a strong sense that the platform now leans heavily on AI to make these calls. Vinted's response is the standard one: if you think a decision was wrong, appeal, and a member of the support team will review it.

That's fine as far as it goes. But it answers the wrong question. The issue isn't whether Vinted has an appeals process. It's that good users are being forced to use it, by a system whose logic they never had the chance to understand, follow or trust.

A trust story before it's a technology story

Trustworthy AI isn't something you can simply declare. As I presented in a recent article, the EU's Ethics Guidelines for Trustworthy AI set out four requirements worth applying here: human agency and oversight, transparency, fairness, and accountability. These aren't abstract ideals, they're testable. Did the user get a meaningful explanation? Was there a chance for a human to intervene before the consequences landed? Could the person understand, in plain terms, what they'd apparently done wrong? 

On the evidence of these Vinted stories, the answer looks like no. Sellers describe being banned with no warning, only discovering a possible reason after the fact, if at all. Automated moderation isn't the problem here; large platforms need automation to operate at scale. The problem is a system built to catch bad actors without the transparency and oversight that protects everyone else caught in the net.

The UK Post Office Horizon scandal offers a useful, if extreme, parallel: people held responsible for decisions made by a system they had no way to interrogate or challenge. Vinted's bans are nowhere near as severe, but the underlying failure has the same shape: a consequential automated decision with no visibility into why, and recourse only after the harm is done.

Applying the PPI framework

I often see user experience problems as being related to one or more of three categories: Proposition, Process or Interface.

Proposition - Are we offering the right thing to the user?

Process - Is the journey, workflow or service process sensible?

Interface - Is the touchpoint itself usable and accessible?

This isn't a proposition problem. People love what Vinted offers, and nobody in these stories is complaining about the core idea. It isn't really an interface problem either; nobody is struggling to list an item or find the sell button.

This is a process problem. The process here isn't just "list an item, get paid." It includes the invisible layer behind that: how the platform decides who to trust, what triggers a suspension, and what happens once that trigger fires. That process now appears automated, opaque and one-directional. Users don't know the rules they're being judged against, get no warning before action is taken, and carry the full burden of noticing, appealing and waiting it out, sometimes with funds frozen in the meantime.

Scale raises the bar, it doesn't lower it

There's a temptation to treat "we use AI to detect bad actors at scale" as a self-justifying explanation. But scale is exactly why the process design obligation is higher, not lower. A wrong call from a human moderator affects one case, and there's someone to explain it. A systematically flawed automated call can affect thousands of good users at once, often with nobody able to fully explain why. This pattern, automated enforcement with no transparency and a bolted-on appeals process, is becoming a defining UX failure of AI-driven platforms.

It's also worth remembering that automated decisions like these don't only cost trust. Users increasingly have routes, including data access and automated-decision rights under data protection law, to challenge exactly this kind of process. Get the process wrong, and the cost saved by automating support could resurface elsewhere.

What good looks like

None of this requires abandoning automation. It requires designing the moderation process with the same care as the checkout flow. In practice that means keeping a person in the loop at two points, not one. First, on Vinted's side: someone who can sense check a flagged account before action is taken, perhaps seeing a long track record, a low-value item, a first offence, and decide the automated flag doesn't warrant a ban. Second, on the seller's side: a chance to see the specific issue and put it right before the consequence lands, not after. Other core practices to maintain trust include:

• Warn before acting, wherever the risk allows it
• Explain the actual reason, in plain language, not a generic policy reference
• Give users a chance to correct the issue before feeling the consequence 
• Make appeals fast and proportionate to what's at stake
• Measure false positive rates and treat a rising rate as a warning sign to fix, not a routine cost of running at scale

Every one of these is a process decision, not a technology limitation. AI can do the flagging. Humans still need to design what happens next.

Vinted's proposition is sound and its interface works. The failure sits squarely in the process, in a system built for efficiency at the platform's end without enough thought for understanding the wider user experience. That's the pattern to watch for as more platforms hand consequential decisions to AI: get the process design wrong, and even the best users end up as headlines.

Want to Go Further?

This follows on from recent articles on AI ethics and the EU's Trustworthy AI guidelines and the ISO RIA Systems framework. 

User Vision is developing a professional training course on UX and AI later this year, exploring the frameworks, methods and practical skills needed to design AI systems that are usable, trustworthy and ethical. The course draws on the work of the  International UX Qualification Board (UXQB)(this will open in a new window), of which Chris Rourke serves as a UK National Expert and member of the curriculum working group. 

If you wish to be notified when the course becomes available, to discuss an AI UX audit for your own products, or simply to explore how these principles apply to your organisation, please get in touch.