In-car user testing – lessons learned

From websites to applications, to sat navs, all user interfaces require testing to ensure they meet the requirements of the end user. You can use the lab or you can, and should where possible, test in situ. No more so was this the case than with my recent testing of a new sat nav system on the roads of Edinburgh. Here I’ve shared some of my learnings for anyone else considering such an exercise.

A bit of background

Sat navs are supposed to help us – but research shows that poorly designed sat nav systems can put drivers in danger, by distracting drivers and taking their attention away from, well, controlling a multi-ton vehicle.

For sat navs to be successful, useful items, they require an intuitive design, with clear instructions and most importantly, ease of use. From fine tuning the timing of instructions, choosing a clear and inoffensive tone of voice, to creating a visual language that supports understanding with a quick glance at the screen, the design of a sat nav system is no easy feat.

Similarly, conducting user testing in a moving vehicle adds some complications that are not present during our usual lab-based sessions.

Lessons learnt: Preparation is key to a smooth journey (metaphorically and literally!)

1. The technical set-up

Know your kit – I used a camera mounted to the headrest and, as I’d be driving around with different users all day, this camera required a large SD card as well as multiple battery changes. Challenge number one.

Challenge number 2: The tablet I tested the sat nav software on needed to be kept charging and to be connected to Wi-Fi or an internet hotspot to work.

Lesson learnt – Figure out and test your required kit.

2. Planning ahead

Always, always, plan – we needed to take a route that would test the features of the sat nav as fully as possible, guiding our participants through things like roundabouts, school zones and quick, consecutive turns.

Finding suitable routes that met all these requirements meant a lot of driving around the city and mapping of possible routes.

This isn’t as easy as it sounds, as navigation systems will generally take you by the quickest and simplest route by default. I also had to incorporate a missed turn, adding and deleting additional destinations and account for the resulting route recalibrations.

Google Street View was a time-saver here, allowing me to check out any unfamiliar sections of the route without actually driving around Edinburgh. Ideally, I also wanted a destination that the local participants wouldn’t be likely to already know their way to.

Lesson learnt – Test drive the route to make sure the tech and the route work as planned.

3. Parking

Paying for parking between participants added another element to remember, and hassle free all day parking would definitely be something I would look for next time, as well as large and easy spaces that don’t require participants to show-off their reverse-parking skills at the end of their session.

Lesson learnt – Figure out what you’re doing between sessions and where can you park. Investigate available spaces and apps that allow for quick parking payment.

4. Be prepared

Note taking in a moving vehicle is not the same as note taking in a lab. I repeat: it is not the same.

Watching the road, listening to the sat nav, being aware of what the participant is doing/saying, making sure the camera is still recording and the tablet is still connected to the hotspot doesn’t leave a lot of time for re-shuffling your papers. Make sure everything is completely ready to go and as simple to note take as possible; clip-boards are your best friend.

Consider taking a colleague along to take notes for you. However, adding more people to a confined space can also add to the respondents’ anxiety levels.

The sheets I’d prepared in advance for each participant were essential, and the more that can go on these sheets in advance the better. I would recommend separate pages for metrics and qualitative observations to make things easier both at the time and for reporting.

Lesson learnt – Investigate note taking methods for different situations.

5. Allow plenty of time and be flexible

The thing with this type of testing was that it was not predictable, and was not always controllable. No two sessions were quite the same.

We mostly had our timings dictated to us by the device and the traffic, which varied things throughout the day. Of course it was possible to occasionally pull over to further probe about something, but as it interrupted the flow it was preferable not to.

Sometimes the participant would take a wrong turn by mistake, so your re-routing task would jump the queue, sometimes the sat nav’s instructions would interrupt the participant’s answer to your question. Some participants were comfortable answering questions while driving, while others weren’t.

It was important to stay on our toes and go with the flow, while being aware of which tasks had been completed and which hadn’t. Your test plan must be flexible.

Lesson learnt – Allow for additional time. Testing in situ is not like in a lab. Different factors will throw your timescales off so always allow for more time than normal.

6. Safety first

Like I said, no two sessions were quite the same and it was important to keep an awareness of each participant’s comfort level throughout this testing, as it did have the potential to be a stressful and thus hazardous situation.

I conducted about half an hour of stationary tasks on the sat nav before setting off, made sure that each participant was fully aware that they could pull over at any time, and that first and foremost their task was safely driving the car. I didn’t want them to feel they had to interact with the sat nav while driving if it wasn’t something they would normally do or felt was safe.

The driving tasks were kept simple, such as muting the instructions and finding out an ETA. Mainly the driving portion was about testing the navigation instructions themselves.

Lesson learnt – You are not in the controlled environment that the lab provides, so make sure you have covered as many eventualities as you can think of.

7. Look after your participants

We always go to lengths to ensure our participants know that we are testing the product, not them, but let’s face it, this situation is more flustering than most and required an especially calm and reassuring bedside manner.

Your participants will not be used to the biting point of the hire car, so they’ll stall and rev the engine too much. Some of them will struggle to reverse-park and probably brake too harshly sometimes.

They need reassured that this is ok, and that it’s also ok that they took a wrong turn/misinterpreted the guidance/nearly went the wrong way down a one-way street.

If remote observation is possible then this would be highly preferable to keep the participant relaxed and open, nobody likes an audience when they’re struggling with something.

The balance here is that it would be great to have one person to moderate and one to take notes, as there is a lot going on. Also, make sure your note-taker in the back is not catching your participant’s eye in the rear-view mirror.

Lesson learnt – Testing is already a strange situation for your respondents. This situation is even stranger and potentially more stressful. How you control the situation and reassure the respondent is vital.

Conclusion

User testing in a moving vehicle was certainly a challenge but every second spent on planning and organisation was totally worth it for the insights.

At the end of the day, it’s always better to evaluate things in as life-like a scenario as possible, and this was no exception. Carrying out a pre-test of your test plan is vital. Grab one of your colleagues and hit the road. Do not leave things until the day.

What would testing or user research in situ look like for you? We’d love to hear from you to discuss ideas.

@HannahMunr0(this will open in a new window) @UserVision(this will open in a new window)

hello@uservision.co.uk