During my Outreachy internship with GNOME, Tamnjong Larry Tabeh and I conducted user research exercises under the inspiring mentorship of Allan Day and Aryan Kaushik.

In this blog post, I’ll discuss the usability test we conducted for GNOME’s Files, also known as Nautilus.

This blog post will introduce the study, outline our methodology, and present our key findings from the usability test. I’ve also attached a downloadable report at the end of this blogpost that discusses (in detail) our observations and recommendation(s) for each task performed in the usability test.

Without further ado, let’s jump right in!

1. Introduction

Files is the default file manager of the GNOME desktop. It provides a simple and integrated way of managing files when running a Linux-based OS by supporting all the basic functions of a file manager and more.

With recent GNOME releases introducing significant changes to the Files user experience, and more improvements planned for subsequent releases, the design team wanted to assess the effectiveness of these updates and learn more about other aspects of the user experience.

To support these efforts, we executed a user research project to identify areas for improvement, and gather actionable insights from observed user behaviours that can inform design decisions when addressing identified issues.

1.1. Research Goals

Our research goals were to:

- Assess the effectiveness of the new menu structure and the discoverability of the following menu items:

1. 1. Icon Size editors
  2. Properties
  3. Select All
  4. Undo/Redo
  5. Sort
  6. Open Item Location
  7. Show Hidden Files
  8. Add To Bookmark

- Evaluate the ease of use of Files’s Search feature, and the intuitiveness of its Search Filters.

- Investigate the extent to which any difficulty experienced when right-clicking an empty space in List View impacts the user experience when accessing a folder context-menu.

1.2. Research Questions

Upon completion of the study, we wanted to be able to answer the following questions:

- Menu Structure

1. 1. Is the current organization of the menus effective?
  2. Can people find the buttons they need for basic tasks when they need them?

- Search Experience

1. 1. Do people understand how to search in Files?
  2. Do people understand the search filters and how to use them?
  3. Are the search filters effective for their context of use?

- List View Layout

1. 1. Is it challenging for people to access the folder context menu in list view when they have a lot of files?
  2. Does the current design meet user expectations when accessing folder context menu in list view?

2. Study Design

2.1. Approach

To answer our research questions, we opted for a moderated task-based usability test. This approach meant that we could simulate typical usage conditions and observe participants interact with Files. This made it easy for us to identify pain-points and gaps in the specific aspects of the Files user experience that we were interested in, and allowed us to engage participants in discussions that deepened our understanding of the challenges they experienced with Files.

To plan the study, we started by defining the ideal participant for our research goals. Next, we established an optimal sequence for the tasks we wanted participants to perform, then crafted a scenario for each, after which we designed the testing environment. Then concluded preparations with a pilot test to identify weaknesses in the study plan and implement revisions where necessary before testing with recruited participants.

2.2. Recruitment Criteria

To generate the data we needed, we had to observe individuals who were unfamiliar with the Files menu structure. This requirement was crucial, as previous use of Files could influence a participant’s interactions, which would have made it difficult for us to discern valid usability issues from their interactions.

We also needed participants to be able to perform basic computing tasks independently: tasks like navigating software and managing files on their computer. This proficiency was important to ensuring that any challenges observed during the study were specifically related to the Files user experience, rather than stemming from a lack of general computer skills.

Therefore, we defined our recruitment criteria as follows:

1. Has never used GNOME prior to their usability test session.
2. Is able to use a computer moderately well.

2.3. Testing Environment

During testing, participants interacted with development versions of Files, specifically, versions 47.rc-7925df1ba and 47.rc-3faeec25e. Both versions were the latest available at the time of testing and had identical implementations of the features we were targeting.

To elicit natural interactions from the participants, we enhanced the testing environment with a selection of files and folders that were strategically organized, named, and hidden, to create states in Files that encouraged and facilitated the tasks we planned to observe.

3. Participant Overview

We recruited and tested with six first-time GNOME users, aged twenty-one to forty-seven, from diverse backgrounds, with varying levels of computer expertise. This diversity in the sample helped us keep our findings inclusive by ensuring that we considered a broad range of experiences in the usability test.

Although the majority of the participants reported current use of Windows 11 as shown below, a few also reported previous use of macOS and earlier versions of Windows OS.

4. Methodology

For this usability test:

- We conducted in-person usability tests with six computer users who met our selection criteria.

- The moderating researcher followed a test script and concluded each session with a brief, semi-structured interview.

- Participants attempted eleven tasks in the following order:

1. 1. Change the icon size
  2. Find the size of a folder with Properties
  3. Select all files in a folder with “Select All”
  4. Undo an action with the “Undo” button
  5. Change the sort order
  6. Change Files display from grid view to list view
  7. Create a new folder while in list view
  8. Find a file using the search feature, with filters
  9. Go to a file’s location from search results with “Open Item Location”
  10. Reveal hidden items in a folder with “Show Hidden Files”
  11. Add a folder to the sidebar with “Add to Bookmarks”

- Participants were encouraged to continuously think aloud while the performing tasks and each session lasted at least 40 minutes.

- All sessions were recorded with participant consent and were later transcribed for analysis.

- To analyze the collected data, we summarized the participants’ experiences for each task using Jim Hall’s Heat Map technique and synthesized findings from our session notes through Thematic Analysis.

5. Usability Test Result

Applying Jim Hall’s Heat Map technique we summarized the observed experience for all tasks performed in the usability test. The heatmap below shows the completion rate for each task and the level of difficulty experienced by participants when performing them.

The tasks are in rows and participants are represented in columns. The cell where a row (Task) i...