Data Driven Design: End-User Testing for Authentic Feedback
I love data and feedback metrics in all shapes and forms. It started out by innocently logging my runs in the Nike Run App to track mileage, then I upgraded to a run watch and it grew a little more obsessive to my pouring over the data and drilling down into each detail I could to identify patterns and make little tweaks to improve performance. I was eager for each run so I could work on one aspect or another and then get giddy as I ran the numbers afterwards - did I improve? If not, why? Like I said…a bit obsessive!
So, there’s no surprise that I get this same thrill when it comes to ID work - I crave making data-informed decisions about course design. However, because of the types of projects I tend to work on, it can be challenging to get authentic student data early enough in the design process to make informed decisions about the learner’s needs. These courses are often brand new, so there is no previous data to lean on.
For the first few years of my role, I learned to be ok with not having student data. Instead, I researched best practices a lot, and relied on my teaching/student experiences and the expertise of colleagues and peers to garner all the insights I could. It wasn’t until I took a course in UX Design and Research Practices (thank you Coursera!) that I saw an immediate way to get data without changing our design process too much: adopt end-user product testing methods to our prototype module.
Normally at the prototype stage, we validate the design with the project stakeholders, but doing so with impartial learners is next level. End-user testing affords better assurance that the course truly aligns with what learners want and need from their learning experience. And testing at the prototype stage is key - we have the agility to accommodate feedback without being too costly and holding up the project timeline.
So how do I incorporate user-testing into my projects without driving my Project Managers crazy? Here is my tried-and-true efficient process:
- Pitch the idea to the client - they ultimately need to provide the end users, which will take a bit of effort on their side. I pitch this early, while we’re still designing the prototype, so they have time to recruit.
- Recruit participants - I share a recruitment letter (see attached template) with clients that they can send out that outlines the task requirements and time commitment. I ask for 4-7 participants (it’s a small sample size, but I don’t have a significant amount of time, so a few is better than none). I encourage a range in the demographic, though sometimes that tough, to account for the range of potential learners. It usually takes about a week to recruit participants.
- Have participants experience the prototype and complete a questionnaire - I communicate the expectations for this stage in an email and share the questionnaire out using a Google Form (free and easy to aggregate data). In this email I also schedule the post-questionnaire interview.
- Post-questionnaire interview - I meet with each participant individually (or in groups if I have more participants) so participants can elaborate on their answers and we can dig into the nuances of their feedback. These meetings range from 15-30 minutes, depending on the size of the prototype. I ask the participant if I can record the call, so I have a record of the conversation if I need to revisit it.
- Share feedback with client - after all interviews, I compile the feedback and look for opportunities to improve the prototype design, which I share with the client in a brief report and review with them in a call. I make recommendations on what we should adapt and get the client’s approval.
- Make changes in the prototype - we make the changes and get the client to review them before continuing on with the design and development of the rest of the course.
Because of scope, timelines, and restrictions with our clients, I don’t have the opportunity to do end-user testing on all of my projects, but the ones where I have been able to - wow! While it does add an extra week or two to our project timeline (steps 3-6), it is hugely insightful and has resulted in courses that meet learner needs and increases the buy-in and anticipation for the new course to be finished and become fully available to participants and their peers.
Here are the templates I use for the user-testing, if you’re interesting in trying it out for yourself:
- Recruitment letter
- Questionnaire
- Post-questionnaire interview script and questions
Do you have questions about the process? Have you tried this process or something similar? Share your thoughts in the comments. We’d love to hear from you!