Prioritizing Analysis: Do We Take Enough Time to Analyze and Process Data from User Research?
“I never go back to watch the videos.” For UX Researchers, that’s a harsh statement. Still, you might have come across such statements from time to time. Andrea Lewis, Director of User Experience at GoodHabitz and former Global Director of UX Research at Adidas, has heard such statements several times and describes them as a naughty confession or an exasperated expression for busy researchers.
The analysis stage in UX Research gets shortened or eventually completely skipped for some research projects. Maybe that sounds a bit drastic, but it is undeniable that the analysis, compared to other steps in the UX research process, like, for example, the fieldwork or the design, is a step that many researchers wish to have more time for (see dscout study).
We spoke with Andrea and took a closer look at this phenomenon. In this article, we’ll discuss the risks of skipping the synthesis in user research, explore four approaches, and discuss how we, UX Researchers, can prioritize analysis.
- Is there really no time for UX research analysis?
- The risks of skipping the analysis (or the power of making analysis a priority)
- Create room and capacities for research analysis
- Way 1: Involve people in the analysis
- Way 2: Use digital helpers
- Way 3: Include buffer in planning
- Way 4: Find the right balance based on the project’s impact
- Summary
Is there really no time for UX research analysis?
Let's start with where we currently stand. In her practice, Andrea noticed a pattern around the urge to complete the analysis stage of UX research rather quickly, and eventually, research gets a bit squished in general.
Why does this happen at all? It is a matter of insufficient time, but maybe that’s too oversimplified. Here are three ideas why this might happen:
Other steps of a research study can't get skipped as easily: You have to recruit participants. You have to conduct sessions. You have to share a presentation. Data analysis is also essential, but it often doesn’t include others or has a specific deliverable expected, so it might be easier to take a shortcut compared to other stages.
The analysis is hidden compared to other stages: Only a few people can see the analysis. Also, for presentations, oftentimes, this step doesn’t get too much awareness as everyone wants to dive into the outcomes.
Analysis in quantitative research is more interesting for some audiences: Many people, including typical user research stakeholders, are more used to interrogating statistics than dealing with qualitative insights. Qualitative insights are probably not as easy to question when you want to learn if something is real.
Besides that, there is pressure to deliver outcomes faster, and sometimes you have to make tradeoffs in the name of time. Again, depending on your research setting, this can look differently:
Agency setting: When working with clients, you may have more control over building project plans that ensure analysis time. But at the same time, planning too much time for analysis can scare some away. Also, new clients and internal stakeholders may question the time for analysis if they haven’t already seen the value it brings.
Embedded Teams: From her experience, Andrea noticed more pressure on quick outcome delivery for embedded teams. Being embedded in a multidisciplinary team may not allow the researcher to step back for a week to analyze the data. There are daily obligations and meetings. A considerable part of the role is about evangelizing User Research, so stepping back conflicts with that mission that requires involvement and showing up. Working in quiet is something that often needs to be justified.
So creating the time for researchers to analyze the data can be an incredible challenge, especially for internal teams. However, finding time for analysis when working on discovery studies is easier, based on her experience.
Understanding the potential impact of this time pressure is essential: it supports a tendency to focus on more minor, more consumable insights instead of more significant qualitative projects. With those, you can meet expectations more quickly and don’t have to step back too much, but at the same time, you miss the moving insights that lie deeper.
The risks of skipping the analysis (or the power of making analysis a priority)
When you have been working on a complicated topic, you probably experienced the need to work through it thoroughly. You need to spend time with the data to understand it.
Surprising, well-crafted, and evidence-backed insights don’t appear automatically after conducting research sessions. Whatever we put in a research report has to come from some analytical process and not from what you remember the strongest, heard last or confirms an existing hypothesis (the typical research biases to watch out for).
When we rush the analysis stage, in Andrea’s opinion, we’re missing the “gems, the hidden layers” and staying quiet surface with what we can learn. Beyond missing out on understanding the gathered data to its full extent, this can lead to stakeholders dismissing qualitative data because of its surface level. So, when skipping the analysis, we miss the most critical step: Making sense of the data, digesting, and processing the research. It is the only way to render meaning and useful insight from data.
In our discussion, Andrea returns to the question of what research is, if not careful data analysis. Why do we even conduct research, and spend time and money on planning, recruiting, and conducting sessions, when we are rushing the analysis? Or, as Nikki Anderson-Stanier states it here:
„However, we can't entirely skip this step. If we do, we might as well not do the research in the first place.“
Anything else will lead to biased or poor results, focused on anecdotal snippets. If we’re skipping the analysis, we’re skipping the crucial step of processing the data and making sense of what you gathered and the chance to come to trustworthy insights. It makes a fundamental difference to find out if something from the sessions stuck because it resonated with the researcher or was salient in the data. It’s the crucial and magic stage, where we go deeper and see patterns that could matter.
Of course, sometimes, you must rely on initial conversations, recalls, or notes. But we must recreate the time to analyze data more thoroughly whenever possible. Because only then can you get clarity on all the data.
Returning to our introduction: Going through video recordings of sessions helps to be careful in the analysis and supports researchers in looking out for researcher and observer biases.
Create room and capacities for research analysis
So, what can we do about it? A careful analysis is the most influential stage in the research process. But the practice shows that it gets rushed or even skipped due to existing time pressure, daily obligations, and other factors.
With Andrea, we discussed how we could make space in the dynamic research workflow for a thorough analysis. She has faced the same struggles in her research projects and tries to support her research team in finding ways “to cheat the time system” in the name of reliable results. Here are her tips:
Way 1: Involve people in the analysis
“Have an analysis party” - or simply find ways to involve colleagues in the research analysis. People who you might want to include are UX designers, product owners, engineers, or other roles who share an interest in the research journey.
Involving people in this stage comes with many benefits:
First, It can support you in finding time for the analysis.
It shows others what this often invisible stage looks like.
You bring everyone closer to the actual data.
You advance the analysis as more perspectives are considered.
Final outcomes will probably be better received as people already saw some “proof”.
It’s a way to peer review the first analysis and avoid being led by typical biases. It’ll challenge your assumptions.
Finally, you create an understanding of the needed time to make sense of collected research data.
Andrea organized such analysis parties, invited people from a multidisciplinary team, split up observation notes and recordings, and let everyone present the focused results.
When there’s no multidisciplinary team around you to participate, you can find shorter and simpler ways to involve stakeholders. For example, you could prepare verbatims as a researcher and make them slip into the shoes of one of the participants for the analysis. Or you could reach out to others in your company who might be willing to help and dive into actual research material. Andrea mentions that she had good experiences involving junior researchers and interns from other teams to accelerate the analysis phase and offer them an interesting way to learn about users, research, and analysis.
But of course, even after involving multiple people in the analysis, the researcher needs to synthesize the data, summarize what was learned, and bring everything together in a final insight or report.
Way 2: Use digital helpers
Understanding the nuances and complexities of the human experience can’t be replaced by machines. Still, there are tasks where you can get support and therefore create space for other jobs. Digital tools like Condens, for example, can streamline your research analysis. This way, you can focus on what can’t be done by tools and hopefully get supported in creating space for analyzing thoughtfully.
From Andrea’s experience, areas, where tools can be especially valuable during research analysis are: transcription, creating research highlights, applying tags to raw data, building a shared database, and offering filter options to look at collected data.
With a tool setup that supports your research analysis, Andrea mentions that you’ll be faster as there are no more copying data snippets from A to B, and being able to work with different file types on the same level, for example. So creating an environment with the right tools is another way that might help you create capacities for research analysis. Andrea states:
„Tools help us to structure our work, and I think the best part of the tools I'm seeing now is that they make the analysis phase less overwhelming, so people are likely to embrace it.“
For research analysis, digital tools can help you with the following:
Structuring, recalling, organizing information
Showing you where you’ve skipped steps in understanding the participant's story
Make the analysis phase less overwhelming
Finding affinities in unstructured data
Accelerate the research work
At the same time, she states that the tools can never help you to understand. Instead, you must spend time with the information, go back to raw data, and process transcripts and other data.
Even for solo researchers, tools can already support the research analysis, especially if you’re not enjoying going through piles of notes:
„But if indeed it's just you and a tool, I think it's better than you alone. I've walked in on researchers with the ground filled with post-its and big piles. Some people love getting deep into the analysis. But I think these tools are a godsend, especially for others who don't necessarily love that or work that way.“
Way 3: Include buffer in planning
Whenever possible, include capacities for analyzing the data as soon as you create research plans already. Make working with transcripts, recordings, and raw data an individual step of the research process that needs extra time. This will make this step more visible and, over time, help you show the value of taking the time for a thorough analysis.A user research plan typically contains particular elements around the research subject and the project’s objectives and is often created at the beginning of a new project. But what to put in a research plan, and how do researchers actually use research plans in practice? Get valuable expert tips and a template for your next user research plan.
Way 4: Find the right balance based on the project’s impact
Having enough time and capacity for proper research analysis and synthesis is the goal, but probably that’s not possible for every single research project. You are working on multiple studies at the same time. You have other daily work obligations, so even when involving people (way 1), making use of digital helpers (way 2), and including analysis time in planning (way 3), it’s likely that there’s sometimes just no time for analysis. So prioritizing and compromising on where an in-depth synthesis is undebatable and where there might be a little room to shorten (not skip) this stage is another way.
Ask yourself where it’s vital to get to the bottom of the gathered data. For example, when working on illness detection trackers, you should probably prioritize the analysis, as insights can affect some people's health. But for other products or other areas, you will probably also research low-impact things where you might be able to shorten the synthesis time a little bit.
Or even better: find a way to shorten the time of gathering data or the amount of data you want to collect (without giving up on the quality of data) but still have enough time to make sense of it. Focus the analysis intensity based on the estimated impact:
„We're creating things for people to interact with and using our ethical discretion, where we know rigor is required versus less than perfect. Not all products require that. Not all services require that. Some are life important and serious, so I gauge my intensity based on what I'm working on.“
So be careful to be scrappy in the right areas of the research process. If you want to dive deeper here, check out this video by Aonatalks:
/f/99166/792x401/278cb97435/screenshot-2023-01-24-at-10-49-58.png)
Summary
When it comes to UX research, it can happen that you’re tempted to shorten the analysis stage or even feel external pressure to skip this stage as outcomes are expected quickly. But this is not a compromise that sometimes has to be accepted. When skipping the analysis, we miss the most critical step: Making sense of the data, digesting, and processing the research. It is the only way to render meaning and valuable insight from data and discover moving insights that lie deeper. So, here are four tips to make analysis in UX Research a priority:
Involve people in the analysis process
Make use of tools to support your analysis
Include enough buffer in the research plan
Find the right balance based on the project’s impact
Andrea Lewis is currently the Director of User Experience at GoodHabitz, leading a team of product designers, UX designers, service designers, and UX and user researchers. Prior to this role, Andrea was the Global Director of User Experience Research at Adidas, and she led various user-centered design leadership roles across the UK government, notably working for the Cabinet Office and GDS for many years. During the early years of the COVID pandemic, she led the user research team for Universal Credit and legacy benefit solutions for the UK’s Department of Welfare and Pensions.
She believes that user experience strategies and relying on insights directly from those with a lived experience are vital to creating products and services that are helpful and useful in the modern world.
Sina is a UX Researcher and Content Strategist at Condens. With a background in social media strategy and digital transformation, she learned how to engage with people in developing products and services and why putting the users in the center will lead to more substantial outcomes and superior user experiences. Sina is excited to drive user research at Condens and to enable better user research for others at the same time by bringing their perspectives into the product development of Condens and by creating valuable content for UX Researchers.
