Key Experience Indicators - focusing on quality in user experience

Human beings, typically referred to as users in a digital context, are quite variable creatures, which creates a persistent challenge in measuring the quality of the experiences they have with software or systems.

Quality, by its very nature, is about comparing things, especially where one of those things is the way things should be. The world of quality relies on having an ideal to measure something against. This could be how long it takes to do something, how accurate or strong something is, and much more. We like to compare things to perfection and see how they stack up.

With user experience, the challenge is that there is no single, absolute perfection. Each person’s experience is subjective and influenced by their own expectations, skills and previous experience. To make matters worse we are pre-configured with another confounding factor – Opinions - which can make it even harder to determine a common view of the overall experience quality.

Even our own experiences with the same task change over time. Think of the first time you used a supermarket self-checkout kiosk. You had to think carefully and perhaps made errors such as putting items in the wrong area after scanning, struggling with pricing loose vegetables, or figuring out how to get your store loyalty points. After doing it a few times you’ve built a mental model and can quickly check out. Same person, same task but two different experiences.

Measuring the user experience quality is less exact than measuring the quality of software or hardware, but it can be done by looking at the right metrics.

Key Experience Indicators

Fortunately, we can gather data that gives us a more balanced and objective view of the collective user experience. These key experience indicators are proxy measures for the actual experience for an individual user.

Key Experience Indicators (KEIs) provide a quantitative score of a specific, important, and actionable phenomenon related to using a product or service. They have been developed from projects and products aiming to have reliable, repeatable measures of the user experience.

It’s tempting to equate them to Key Performance Indicators (KPIs) which can be applied to anything from mechanical strength to a person’s performance on a job. KPIs like a material strength or a vehicle’s speed can often be directly measured. KEIs however differ because they typically require some indirect, proxy measures which serve as indicators of an assumed individual’s experience.

Several user experience professionals have applied the concept of KEIs, and there are useful predecessors such as Google’s HEART Framework(this will open in a new window). KEIs were pioneered by Tomer Sharon, a user researcher with Goldman Sachs who proposes several benefits to establishing clear, measurable KEIS, including:

1. Providing experience information to decision makers
2. Identifying strengths and weaknesses of a product
3. Improving the product/market fit
4. Providing a baseline to improve from

Through his research Sharon has established 16 KEIs useful measures of the user experience which he outlines in an article(this will open in a new window) . The KEIs include:

• Time on task – how long someone takes to complete something
• Lostness – which measures the efficiency of using a product, based on the optimal and the actual number of steps a user takes to complete a task.
• Adoption rate
• Retention rate - the percentage of retained users over time
• Satisfaction score

Interestingly, one that is not included is the Net Promoter Score, a metric used in many commercial enterprises. NPS has a questionable feature of asking people to predict their future behaviour, specifically whether they would recommend a certain company or service. Although the NPS is a well-established and widely-reported metric, it is based on a strange calculation and relies on each person predicting their future, something we don’t have a great track record of doing.

Deciding the KEIs for your organisation

Some KEIs will be more relevant to your organisation than others so choose them carefully based on what data you can capture and whether you are evaluating a software system, a product or an end to end journey. The more relevant a KEI is to your business, the more likely it is to be welcomed and adopted as part of your core business metrics.

You may have different KEIs in addition to those proposed above. So how do you decide the right KEIs for your organisation? The process of actually deciding the KEIs is a valuable exercise in itself since it can gather the views of several disparate stakeholders.

We find it useful to consider a framework with three broad categories:

1. Facility – Can people use the product/service/feature?
2. Engagement – Do people use the product/service/feature?
3. Satisfaction – How do people feel about using the product/service/feature?

Within each of these, there should be some measurable KEIs. For instance, task success rate would relate to Facility, while adoption rate is more related to Engagement.

You can also apply a Goals, Signals, and Metrics process, inherited from the HEART framework, to decide what are the measurable metrics that will serve as your best KEIs.

Track your progress

The best use of KEIs is to track them over time. A single snapshot of data is unlikely to be enlightening because it may not directly point to the cause of an apparent issue. Humans, armed with our opinions and vagaries of company politics can attribute perceived issues to the wrong underlying problem.

The best approach is to consistently measure the KEIs and be aware of confounding factors that may be affecting the results such as seasonality which may affect the amount or type of traffic a website receives.

Measuring KEIs over time as you make certain changes, such as to a website’s content, will be the best way to identify what is moving the needle in either a better or worse direction. Once you have a hint of where the issue may be, it’s usually a good idea to perform qualitative user experience research such as usability testing or digital diary study research to fully understand the behaviour and attitudes that are driving the indicators.

Just do it

If you’re new to the world of KEIs it may be daunting to determine the right KEIs that fit in the sweet spot of being measurable and relevant to the company goals. Don’t be discouraged if they are not immediately apparent.

Depending on your position and company, it may be best to start small, select 2 or 3 and do them well. Once the system is in place, you have confidence in the underlying measures and you’ve been tracking the KEIs for long enough, share your findings to relevant stakeholders, broaden the conversation and see if other KEIs could be brought onstream.

Just as there is no set of KEIs that works for all companies, there is no universal way to introduce them to an established organisation. You’ll learn lessons along the way as you perfect your KEI set, and you can be confident you are continuously learning more about your audience throughout the process.