The Unintended Consequences of Making Things Easier

Making things easier sounds like an obvious good thing. But when we remove effort from an experience, we don’t always know what else we’re removing with it.

AI scribes are a good example. Already being used by GPs, they can listen to consultations, turn conversations into notes and letters, and reduce a significant amount of administrative work. There are clear benefits to that. But new research from the University of Edinburgh(this will open in a new window) (recently reported on the BBC(this will open in a new window)) suggests we need to look beyond the productivity gains and understand what else changes when we introduce AI into potentially sensitive discussions.

For us, there are three lessons worth paying attention to:

1. Capturing the conversation isn’t the same as capturing the experience

An AI scribe can capture what someone says. But a consultation is about more than words.

A patient’s hesitation, tone of voice, facial expression or emotional state can all provide important context. And the things people don’t say can be as important as things they do.

The research found that AI-generated summaries can prioritise clinical information at the expense of the patient’s wider story and experience of their illness.

That’s an important distinction. Data isn’t the same as insight.

We see this beyond healthcare too. Analytics can tell you what people clicked and surveys can tell you what they said. AI can now summarise thousands of interactions.

However, none of those things automatically tell you what the experience was actually like.

2. The technology can change the behaviour it’s designed to observe

The research also suggests that some patients may be less willing to disclose sensitive information when they know their consultation is being recorded and processed by AI.

So the technology isn’t simply observing the interaction. Its presence has the potential to change the interaction.

This is something we have observed across a range of clients, where the level of user comfort and trust in AI-enabled services is often significantly over-estimated by digital teams.

This can be so easy to overlook when we’re designing or introducing new technology. We tend to think about whether a new feature works as intended, rather than asking what effect it has on people’s behaviour. Will customers still ask for help if they know they’re talking to an AI? Will people provide the same information? Will they trust the service in the same way? Will some people be affected more than others?

These aren’t questions that can be answered by testing whether the technology itself works.

3. Efficiency isn’t necessarily effectiveness

One of the most interesting findings concerns “cognitive offloading” – allowing technology to take on some of the mental effort previously carried by a person.

There are obvious benefits. If a GP spends less time writing notes, they may have more time to concentrate on the patient.

However, the research also highlights concerns that relying on AI-generated notes could affect memory, recall and skill development - a useful reminder that removing effort isn’t always the same as improving an experience.

Sometimes the effort we’re removing is unnecessary friction. Sometimes it is part of the expertise, judgement or engagement that is crucial in making the service work effectively.

What might this mean for you?

As organisations introduce AI and automation, it’s tempting to start with the technology:

What can it do? How much time can it save? How much can we automate?

Those are important questions. But it is equally important to understand what happens when real people encounter the new experience.

• Does it change their behaviour?
• What information might be lost?
• Where does cognitive effort move rather than disappear?
• Does it work equally well for people with different needs and circumstances?
• Ultimately, is the experience actually better?

These questions become particularly important when technology is introduced into services where trust, judgement, communication or human interaction are of fundamental importance.

That’s where research, service design and inclusive design can help.

At User Vision, we help organisations understand the people behind the data, test how new experiences work in the real world, and identify the unintended consequences that aren’t always visible in analytics or performance metrics.

Don’t just measure whether the technology works. Understand what it changes.

If you’re introducing AI or redesigning a digital service, talk to us about how research, design and accessibility can help you understand the impact on the people who rely on it.