User Diary Studies - An Effective Research Method for Evaluating User Behavior Long-Term

User diaries, or user diary studies, are a self-reporting research method. Participants log their experience, behavior, activities, and thoughts over a certain period of time. Hence the name "diary." Researchers collect data about the experience over time, usage of features, habits, and emotions. Researchers can also ask participants to take pictures or videos to better understand the user’s environment. The data collected is primarily qualitative. However, depending on your research goal, you could collect some quantitative data additionally.

It's a discovery phase method: You use it at the beginning of the research process when you are trying to frame and understand user problems. Researchers can use diaries for validation too. This helps when you want to make sure your design works and solves the users' problems. I use diary studies on my current project to understand if content is missing on the beta version we already shipped. More on that later in the article. Diary studies are often conducted after the first phase of interviews or contextual inquiries.

The main advantage of a diary study is the ability to collect data over time. Also known as "longitudinal information." Researchers can understand temporal dynamics: how the user experience might change over time. It's an excellent method to collect information about habits and processes. Additionally, this method helps to understand the events' flow and successions over time.

This method also lets you capture unplanned or hard-to-plan tasks and activities. It is an excellent method to gather data about behaviors or tasks that happen sporadically.

Also, it's possible to learn about tasks or activities that occur at a specific time which can’t be replicated by asking participants to do them in front of you. For example, if you work on a period tracking app, a user diary study over a certain time can cover varying cycles. Every cycle can be different and they can be very unpredictable. So observing users filling in data when the cycle starts might be hard to plan.

A user diary study is a self-reporting method. That means that users will input the data themselves, without a researcher’s help. It is less intrusive than direct observation or over-the-shoulder shadowing. Researchers use it in situations where they might not be able to conduct direct observation. For example, it’s particularly suitable for sensitive data or populations that can't be observed directly. It also eliminates bias during observation. This makes the data collection more natural in its context. Participants can report influential external factors from their natural environment. For example, during a project to monitor cranes on construction sites, crane operators used our tablet app in bright sunlight, dark spaces, and very dusty environments. We found that contrast was super crucial for that app.

The main advantage of user diary studies can also be a significant limitation. This method follows users over some time. If your time for this research is relatively short, it is not a suitable method for your current project. It also requires a lot of commitment from the users. Which often leads to bigger incentives and bigger budgets. In other words, this is not a "quick and dirty" method. People might also forget about completing it. Follow-ups and reminders are essential (more on that in the next part).

Beware that self-reported data may not be 100% accurate and is often biased. People are not always good at reporting their own emotions, for example. If possible, keep your journey task-oriented and factual. Don’t ask people to imagine their future. Don't ask them to remember a far away past. Don't ask things like “how much would you pay to use this tool in the future?” or “what other tools did you install in the last year to accomplish that task?” A year is a long period of time.

Another limitation is missing data. Researchers are not observing participants. The user entries in your diary could miss interesting data users didn’t think about self-reporting. You can mitigate this with accurate instructions and a good kick-off session.

Also, note that you will gather a LOT of data. Prepare yourself for a long analysis session.

As a general rule, investment banks are often a bit behind the curve when it comes to technology. I recently worked on a project updating an internal tool for just such an organization. The platform was intended to help employees from different departments create and follow financial projects. The interface we were tasked to improve was 15 years old and required a complete rebuild!

To understand how people used the existing tool, we conducted user interviews. We collected data on their tasks and activities to ensure we migrated the right content and features.

Since we work in an agile way, we had already created pages and features that were available for users in a beta version. After the interviews, we gave our early adopters access to that beta version.

We then conducted a diary study to understand the user migration and usage of our new version over a month. We wanted to understand a few things:

• How is the adoption going for new users?

• Was there anything that stopped them from doing their daily tasks and activities with the new tool (missing content, features)?

• Were there things we migrated but users couldn't find in the new tool (usability or information architecture issues)?

• Did they have any improvement suggestions?

The diary study started with a one-hour kick-off session with users. We gave users access to the new tool and observed them performing their daily tasks on the latest version.

Next, we explained the concept of the user diary. We asked them to use the new tool on a daily basis and provided an excel sheet to be completed over the next month.

Finally, we explained the trigger: log an entry whenever you needed to do something but could not with the new tool (and had to go back to the old one).

Users logged their data in a shared spreadsheet.

After a month (and some follow-ups), we collected the diaries and merged them into one extensive document. In this document, we added a few extra columns to help us analyze the data:

• Type (usability issue, improvement, new feature, new content, bug): Color-coded for faster identification within the file.

• Gravity (trivial/minor/moderate/major/critical): Helpful for prioritization.

• UXD comment: A column for extra information, like how we could improve it, or "have a follow-up interview with a user for this topic."

• Story: To link to the user story.

• Status: To show the status of the story.

• Tag: The main topic of the entry. This is used for quantitative purposes to see which main issues we need to tackle in the subsequent releases.

Here’s what that looked like for our project on internal tool migration:

All diary entries were compiled and color-coded to prepare research analysis.

Once our data was combined into a single spreadsheet, we went through each diary entry and began our analysis. We discussed each task that users could not complete using the new version of the tool, determined the gravity of the task, and if/how we would incorporate this into the future tool.

Our diary entries were very task-oriented. For example, people reported specific tasks they were trying to accomplish with the search and said precisely who/what and how they were searching for this information. They helped us gather feedback and plan improvements.

This also let us identify usability issues for features or content that might not have been clear in the new tool. For example, if a user logged content as missing, but we already migrated it, we might have an issue.

All that data also helped with training and building our FAQ, which our enterprise users expected as part of change management.

Conclusion

As a result of the user diary study, we were able to ensure the new tool incorporated all the functionality of the original but with the updates that were so desperately needed. The users played an integral role by allowing us to see what components of the tool were essential for them to do their jobs effectively. The diary helped us evaluate what was missing by giving us a sneak peek into their daily tasks.

If you choose to go with an excel sheet, you can use my template here:

This template for user diary studies contains study's instructions and a sheet to collect diary entries. Make a copy and adapt it for your next study.

Open template

The first sheet is the instruction sheet. Put in there all the instructions for your users: when to log, how much information to enter, contact, etc. Then ask your users to log entries into the second sheet, the “Log here”-sheet. This sheet is for data collection. You need to adapt the columns to your specific needs. In the next step, you can build a consolidated sheet with all the data you collected from all the diaries.

• User diary studies are a self-reporting method: users fill in diary entries on their own.

• It is a great tool to collect data over time, if you can’t observe participants directly, for unplanned tasks and tasks that can’t be replicated.

• It requires a time commitment from the participants and the data might not always be perfectly collected.

• There are 3 types of triggers for a diary: event, prompt, and interval.

• Chose the trigger and format upfront.

• When recruiting participants, make them aware of the time it'll take.

• Schedule onboarding sessions and follow-ups if you want the method to work.

• You can use quantitative and qualitative data to analyze your results.

• Schedule follow-up sessions with participants to get more information and thank them.