*** Disclaimer: Some information about participants, data, and the product was modified for confidentiality ***
Overview:
In this project, I was in charge of study design, moderated user testing sessions, analyzed data, generated insights and recommendations, created a report deck, and presented the case to high-level stakeholders.
My Role: UX Researcher
Method: Evaluative Usability Testing
Stakeholder: Etisalat
Timeline: 4 days
Background:
During UAT a delay of 1 second was found when pressing the play button on selected songs. [After benchmarking, the engineering team found out the delay is ــــــ ms when neutralizing all other conditions including phone and network performance]
In this study, our objective is to gauge users' perception of the delay before spending the engineering effort [estimated cost: XX,000 EGP].
Research Brief:
📋 Method & Procedure
1:1 Usability tests & interviews
30-minute in-person sessions
Testing was done on a public beta version of the app
🙎 Participants
20 Participants (Males)
Mix of different professions, ages, and education
Smartphone owners who use music streaming services
Study Planning (1 day)
As part of the briefing meeting, I was told that 1) we need to evaluate if users will notice the delay before investing the engineering effort 2) we have 5 days to reach a verdict before engineering commits to other sized tasks 3)
As a follow-up, I created a research plan with the testing task "Imagine you're at <your favorite location> have the Twist Music app opened to its Home Page, think of a list of five songs then play them on the app"
Probing and interview questions:
Questions
User Testing and Data Analysis (2 days)
I intercepted participants in malls/cafes to participate in the research using screening questions to filter the target users.
I used the moderator guide I prepared (shared with stakeholders as part of the research plan) to moderate the usability sessions. During the sessions, I took notes on users' behavior and comments
After all user testing sessions were finished, I organized the data from the recordings and notes to uncover patterns.
Screen capture from user testing session
Reporting and Presentation (1 day)
I created a report presentation of the research process, including the background, methodology, insights, rationales, quotes, recommendations, and the final verdict regarding the primary research objective.
I walked the stakeholders through the report, highlighted the challenges, and shared my recommendations for next actions.
Research Findings
Participants 25 years and younger were more likely to notice the delay. Follow-up questions show that participants were unsure if the delay was due to app or network performance.
Impact
The research presented here successfully informed the project team and helped prioritize their engineering efforts. Also, it pointed out where to focus their following round of testing to investigate specific usability issues based on my research-reported recommendations.
Challenges
Some situations that arouse throughout the research process
1— Tight time frame:
As video recording was manda
2— Participants' sample is skewed:
As video recording was mandatory, I wasn't able to recruit any female participants. This was anticipated beforehand, but due to the project's tight timeframe, there was no chance to recruit participants ahead of the sessions.
My advice was to still carry out the research even if we ended up not recruiting any female participants (My rationale was: this is better than no research + secondary research shows males are faster with reaction times).
3— Task confusion:
Due to some limitations in the app's library, one user searched for a track that wasn't available and was confused about what they should do next. Instructions were modified to address this issue although it did not occur in the remaining sessions.