User Diary Studies - An Effective Research Method for Evaluating User Behavior Long-Term

User Diary Studies - An Effective Research Method for Evaluating User Behavior Long-Term

July 12, 2022

One interesting UX Research method to keep in your toolbox is user diary studies. This method is beneficial if you aim to collect data over a certain period. It’s also a handy user research method when direct observation is impossible.

In this article, you’ll learn about the basics of user diary studies as a UX research method. You'll see when it works best, and where it has its limitations. I’ll walk you through all the steps to successfully conduct a user diary research study along with a real-world example from a past study. This also includes a template for user diary studies.

What is a user diary? 📔

User diaries, or user diary studies, are a self-reporting research method. Participants log their experience, behavior, activities, and thoughts over a certain period of time. Hence the name "diary." Researchers collect data about the experience over time, usage of features, habits, and emotions. Researchers can also ask participants to take pictures or videos to better understand the user’s environment. The data collected is primarily qualitative. However, depending on your research goal, you could collect some quantitative data additionally.

It's a discovery phase method: You use it at the beginning of the research process when you are trying to frame and understand user problems. Researchers can use diaries for validation too. This helps when you want to make sure your design works and solves the users' problems. I use diary studies on my current project to understand if content is missing on the beta version we already shipped. More on that later in the article. Diary studies are often conducted after the first phase of interviews or contextual inquiries.

Advantages and when to use this method 👍

The main advantage of a diary study is the ability to collect data over time. Also known as "longitudinal information." Researchers can understand temporal dynamics: how the user experience might change over time. It's an excellent method to collect information about habits and processes. Additionally, this method helps to understand the events' flow and successions over time.

This method also lets you capture unplanned or hard-to-plan tasks and activities. It is an excellent method to gather data about behaviors or tasks that happen sporadically.

Also, it's possible to learn about tasks or activities that occur at a specific time which can’t be replicated by asking participants to do them in front of you. For example, if you work on a period tracking app, a user diary study over a certain time can cover varying cycles. Every cycle can be different and they can be very unpredictable. So observing users filling in data when the cycle starts might be hard to plan.

A user diary study is a self-reporting method. That means that users will input the data themselves, without a researcher’s help. It is less intrusive than direct observation or over-the-shoulder shadowing. Researchers use it in situations where they might not be able to conduct direct observation. For example, it’s particularly suitable for sensitive data or populations that can't be observed directly. It also eliminates bias during observation. This makes the data collection more natural in its context. Participants can report influential external factors from their natural environment. For example, during a project to monitor cranes on construction sites, crane operators used our tablet app in bright sunlight, dark spaces, and very dusty environments. We found that contrast was super crucial for that app.

Limitations and Pitfalls of User Diary Studies 🙅

The main advantage of user diary studies can also be a significant limitation. This method follows users over some time. If your time for this research is relatively short, it is not a suitable method for your current project. It also requires a lot of commitment from the users. Which often leads to bigger incentives and bigger budgets. In other words, this is not a "quick and dirty" method. People might also forget about completing it. Follow-ups and reminders are essential (more on that in the next part).

Beware that self-reported data may not be 100% accurate and is often biased. People are not always good at reporting their own emotions, for example. If possible, keep your journey task-oriented and factual. Don’t ask people to imagine their future. Don't ask them to remember a far away past. Don't ask things like “how much would you pay to use this tool in the future?” or “what other tools did you install in the last year to accomplish that task?” A year is a long period of time.

Another limitation is missing data. Researchers are not observing participants. The user entries in your diary could miss interesting data users didn’t think about self-reporting. You can mitigate this with accurate instructions and a good kick-off session.

Also, note that you will gather a LOT of data. Prepare yourself for a long analysis session.

How to run your User Diary Study from A-Z 🎯

Step 1: Prepare the diary study

Like most research methods, the first step is to prepare your study. Here’s how to:

  1. Start with your research goal/question: what are you trying to learn or understand? What data do you want to collect? Good research is about asking the right question, to the right people, at the right time. You need to define a clear scope and user group: who will you send the diary to? Tip: Document the answers in a user research plan.

  2. Then, decide on the trigger: when do people need to log something in the diary? The type of trigger depends on your research question and what you are trying to understand. As mentioned above, this can be an event, a specific interval of time, or a prompt. For example:

    "people log whenever they use a specific tool" (event trigger)

    "people log every week, on Sunday afternoons" (interval of time trigger)

    “people log whenever we send a prompt” - In case the stimulus is a prompt, you need to decide when you want to prompt people. And how you will prompt them.

  3. Also, decide on how long the study will run. You can't expect people to fill in a diary every day for six months. How much data you need depends on your study. But don't be greedy. Do you need THAT much data? Usually, 2 weeks to 2 months is a good time frame. But again, this depends on your type of study.

  4. Next, decide on what to capture and how: this could be for example text, audio, video, or photos. This might influence the tool for the diary. In the end, go for the tool that makes sense for your users. Choosing a tool they are comfortable with can be crucial for the success of the study. Here are a few options:

    • Old school written diaries: cheap, but handwriting can be hard to read

    • Digital diaries in a digital file: like a Word document, or Excel sheets. If they are supposed to add photos, PowerPoint or Keynote could do the trick. You can ask people to fill one slide per trigger.

    • Online survey tools: people fill the same survey, again and again, for each trigger.

    • Communication tools: emails, Slack, WhatsApp, etc. Those tools are great if you need prompts. Exporting data might be tricky though.

    • Some UX research tools offer specific diary study options.

Step 2: Design the study

Diaries often have two parts: instructions and the actual data collection part.

Even if you have a kick-off meeting, reminding users of some information about the diary is essential. Here is a checklist of what you can put in the instructions:

  • the goal of the research to help understand the context
  • when to log (trigger)
  • how long will they need to fill in the diary
  • who and how can they contact in case of questions, issues, etc
  • specific instructions depending on what you collect
  • an example of an entry

With each entry, the diary becomes a more complete collection of the data you are requesting from users. You can have a very open format with one big, generic field where users record an entry however they want. Or, you can have a closed format with predefined, closed-ended questions.

After you’ve created your diary and your instructions (incl. prompts and questions), don't forget to pilot test it. Run a version with colleagues to make sure you didn’t miss any typos. Also, pilot the prompt system if you need one. For example, test the tool you will use to send emails to ask people to complete their entries. Make sure the email arrives and is readable.

Step 3: Recruit the participants

You recruit users for user diary studies like any other UX research study. But, there are a few specific details you need to be careful about when conducting a user diary study:

  • Diary studies require commitment over time; make it clear when you recruit.

  • The higher the commitment, the higher the incentive.

  • You will have dropouts, schedule extra participants.

Step 4: Launch and Follow-up

You designed a very nice diary study. Great. But, you can't just send it to users and expect them to fill it in. To launch a diary study, start with an onboarding session with the users. It can be a 1-on-1 session or a smaller group session. I tend to prefer 1-on-1s to answer specific questions.

Explain what is expected of them and how long the study will run. Also, explain the trigger, especially if it is an event trigger. Answer all the questions they have. More importantly, ensure they know how to contact you if they have more questions.

Go through the diary. If it's a digital file, make sure they can open it. If it is a survey, make sure it works in their browser. The goal of this session is to remove every friction that might happen once they are on their own.

During the study, don't leave participants alone if you want them engaged. Check-in with them and send them reminders, even if it is not a prompt based survey. It can be a short email, or a quick call, the format is up to you. If some people dropped out, follow-ups are a way to try to bring them back and understand why they dropped out.

Step 5: End the study and get the diaries back

Let people know a few days before you will arrive at the end of your study. If the tool wasn't synchronous, like a Word or Excel sheet, it is time to collect the files. Don't forget to back them up somewhere, just in case.

And of course, thank the users for their time and explain how to get the incentive.

Step 6: Analyze the data

I usually do a first run of "quick analysis" by reviewing the files and answers to get a quick idea.

If you use a tool that lets you get the data in real-time, you can start a pre-analysis before the study is completed. Be careful not to draw any conclusions yet, though! However, you can already tag the data, start noticing patterns, etc.

How you analyze the diary's content depends on your research question and scope. Since you’ll get a lot of open content, this will take time. I recommend doing this with your team. I am a big fan of excel sheets for such work.

I always start by cleaning the data. Are there any diaries that have been completed poorly? Remove them if the information quality won't bring anything to the study.

You can make use of dedicated tools for both quantitative and qualitative analysis. Qualitative data (open content, audio transcripts, long texts,...) can be a bit tricky, but there are plenty of methods you can use. I find techniques like thematic analysis and rainbow sheets quite helpful.

Step 7: Follow up with the users

Depending on the study, you want to debrief with participants once you have analyzed the data. Maybe you want to know more about one topic. Perhaps there is a specific entry you don't quite understand. You can schedule short follow-up sessions with the users if needed.

Follow-up sessions are also a good place to thank your participants again. You can also ask them for feedback about the diary, especially if you want to rerun it. Was it too long? Too difficult? Were the questions clear? Did you have some people drop out?

Step 8: Sharing the research results

Following up and finishing the analysis brings you to the next step: sharing the results with your team(s). The format, again, depends on the data collected and the study. It can be a report. Or you could build customer journey maps, empathy maps, etc. The format is up to your specific needs.

Real-world example: User Diary Study for the rebuilding of an internal enterprise tool 💡

As a general rule, investment banks are often a bit behind the curve when it comes to technology. I recently worked on a project updating an internal tool for just such an organization. The platform was intended to help employees from different departments create and follow financial projects. The interface we were tasked to improve was 15 years old and required a complete rebuild!

To understand how people used the existing tool, we conducted user interviews. We collected data on their tasks and activities to ensure we migrated the right content and features.

Since we work in an agile way, we had already created pages and features that were available for users in a beta version. After the interviews, we gave our early adopters access to that beta version.

We then conducted a diary study to understand the user migration and usage of our new version over a month. We wanted to understand a few things:

  • How is the adoption going for new users?

  • Was there anything that stopped them from doing their daily tasks and activities with the new tool (missing content, features)?

  • Were there things we migrated but users couldn't find in the new tool (usability or information architecture issues)?

  • Did they have any improvement suggestions?

The diary study started with a one-hour kick-off session with users. We gave users access to the new tool and observed them performing their daily tasks on the latest version.

Next, we explained the concept of the user diary. We asked them to use the new tool on a daily basis and provided an excel sheet to be completed over the next month.

Finally, we explained the trigger: log an entry whenever you needed to do something but could not with the new tool (and had to go back to the old one).

Users logged their data in a shared spreadsheet.

After a month (and some follow-ups), we collected the diaries and merged them into one extensive document. In this document, we added a few extra columns to help us analyze the data:

  • Type (usability issue, improvement, new feature, new content, bug): Color-coded for faster identification within the file.

  • Gravity (trivial/minor/moderate/major/critical): Helpful for prioritization.

  • UXD comment: A column for extra information, like how we could improve it, or "have a follow-up interview with a user for this topic."

  • Story: To link to the user story.

  • Status: To show the status of the story.

  • Tag: The main topic of the entry. This is used for quantitative purposes to see which main issues we need to tackle in the subsequent releases.

Here’s what that looked like for our project on internal tool migration:

All diary entries were compiled and color-coded to prepare research analysis.

Once our data was combined into a single spreadsheet, we went through each diary entry and began our analysis. We discussed each task that users could not complete using the new version of the tool, determined the gravity of the task, and if/how we would incorporate this into the future tool.

Our diary entries were very task-oriented. For example, people reported specific tasks they were trying to accomplish with the search and said precisely who/what and how they were searching for this information. They helped us gather feedback and plan improvements.

This also let us identify usability issues for features or content that might not have been clear in the new tool. For example, if a user logged content as missing, but we already migrated it, we might have an issue.

All that data also helped with training and building our FAQ, which our enterprise users expected as part of change management.

Conclusion

As a result of the user diary study, we were able to ensure the new tool incorporated all the functionality of the original but with the updates that were so desperately needed. The users played an integral role by allowing us to see what components of the tool were essential for them to do their jobs effectively. The diary helped us evaluate what was missing by giving us a sneak peek into their daily tasks.

Spreadsheet Template 🖇

If you choose to go with an excel sheet, you can use my template here:

The first sheet is the instruction sheet. Put in there all the instructions for your users: when to log, how much information to enter, contact, etc. Then ask your users to log entries into the second sheet, the “Log here”-sheet. This sheet is for data collection. You need to adapt the columns to your specific needs. In the next step, you can build a consolidated sheet with all the data you collected from all the diaries.

Summary

  • User diary studies are a self-reporting method: users fill in diary entries on their own.

  • It is a great tool to collect data over time, if you can’t observe participants directly, for unplanned tasks and tasks that can’t be replicated.

  • It requires a time commitment from the participants and the data might not always be perfectly collected.

  • There are 3 types of triggers for a diary: event, prompt, and interval.

  • Chose the trigger and format upfront.

  • When recruiting participants, make them aware of the time it'll take.

  • Schedule onboarding sessions and follow-ups if you want the method to work.

  • You can use quantitative and qualitative data to analyze your results.

  • Schedule follow-up sessions with participants to get more information and thank them.


About the Author
Stéphanie Walter

Stéphanie is a UX Researcher & Designer based in Luxembourg. She has 12+ years of experience and specializes in enterprise UX and mobile. She teaches, speaks, and writes about design, UX research, accessibility, cognitive biases, design-dev relationship, etc. Besides that Stéphanie enjoys good tea, bike rides, and drawing illustrations. She says about herself: "My D&D alignment is chaotic neutral and I am better at keeping my teammates alive in video games than my plants."

Don't hesitate to reach out to her for questions about research and design!


Want to receive UX Research Meetup & Event guides, new content and other helpful resources directly to your inbox? Enter your email address below!