Pantry pal is an application to assist its users in keeping track of grocery items they have purchased, knowing when those items expire, allowing users to find recipes that use the items they currently have, and creating grocery lists within the application. In doing these things for its users, PantryPal will help reduce food waste and cut down on grocery costs. The goal of this sprint was to conduct a Pilot Test and gain insight from real users.
The PantryPal Usability Engineering team conducted Pilot tests with users on December 4th, 2023 (n = 6). Before testing began, both Usability Engineers worked together to generate a Prototype of the PantryPal application, Fill out an IRB form, create an Informed Consent Form, and generate a Protocol/Script for the study. The Protocol outlines the structure of the study, including what tasks the participants are asked to complete, and what data will be collected/measured during the study. During the planning for the Pilot Test, the UX team determined that the main focus of this study would be comparing efficiency of the PantryPal application to a competitor’s application.
Participants for the study were other students from the Usability Engineering class. The PantyPal UX team posted the requirements for a participant to the class communication channel, and 6 students signed up to participate in the Pilot Test. It was determined that the only requirement for this study was that the participant must be over the age of 18.
For the PantryPal Pilot Test, each participant was asked to complete two tasks. Each task was to be completed twice, once on the PantryPal Prototype, and once on the competitor application, MyPantryTracker. The application that participants started with was varied, in order to help mitigate any bias towards one application or another.
Before the Pilot Test begas, users were read the Informed Consent Form and asked to sign the form to ensure that they understood the study and risks, and were willing to participate. Next, the Facilitator for the test followed the Protocol outline for the test. First, the introduction was read and questions to gather background information were asked. These questions aimed to gather information about the experience that the participant has had with similar applications, and to what scale this application would help them manage their groceries. After the responses were recorded, the Facilitator initiated the tests. For both tasks that users were asked to complete, the primary goal of the task was to test the efficiency of the PantryPal application against a competitor.
With Application 1 displayed on a laptop, the Facilitator read the instructions for Task 1, then asked the participant to complete the task when they were ready. Once the participant began interacting with the application, the facilitator started a stopwatch timer. During the participant’s interaction, the Facilitator took notes on the actions that the participant took to complete the task. When the participant indicated that they believed they had completed the task, the Facilitator stopped the timer, recorded the time, and indicated which, if any, parts of the task the user successfully completed. Once the Facilitator finished recording the results of the task, they opened Application 2 and repeated the exact same process of reading the task, timing the task, and recording the results.
The process for Task 2 is identical to Task 1. For Task 2, the Facilitator re-opened the application 1 and read the instructions for Task 2. Once the participant begins interacting with the application, the Facilitator starts a stopwatch timer and begins to note the steps the participant takes to complete the task. When the participant indicates that they believe they have completed the task, the Facilitator stops and records the time, and makes note of which portions of the task the participant successfully completed. Once the Facilitator has recorded the results, they open Application 2 and repeat the same process of reading the task, timing the task, and recording the results.
Once Task 2 has completed and all information from the tasks has been recorded, the Facilitator begins the Debrief. The debrief aims to gather qualitative feedback from the participant about their experience using both applications. The Facilitator asks participants questions about the features they liked and disliked on each application, as well as which application between the two they preferred. The Facilitator records the participant’s responses to each question, then thanks the participant for their time and the Pilot Test concludes.
After the entire Pilot Test is finished, Facilitators record the data collected, without any identifying information, into a data spreadsheet for reference and analysis later on.
The quantitative findings from Task 1 revealed that 5 out of 6 participants completed the task faster with the PantryPal prototype than with the competitor’s application. Of the participants that completed the task faster on the PantryPal application, the task was completed between 18% to 66% faster than on the competitor’s application. All participants successfully completed the task on both applications. The quantitative findings from Task 2 revealed that 100% of the participants completed the task faster on the PantryPal Prototype than on the competitor’s application. The speed up ranged between 51% to 76%. All participants successfully completed Task 2 on the PantryPal application, however 2/6 participants did not successfully complete Task 2 on the competitor application.
The data collected during debrief revealed that 5/6 participants preferred the PantryPal application, and 5/6 participants felt as though the PantryPal application allowed them to complete the tasks quicker. Qualitative data from the debrief questions revealed that users disliked the competitor’s website because of how difficult it was to use, and liked the PantryPal application because of its simplicity.
The Pilot Test revealed that the PantryPal application is well received by real users. The features of the application matched the participant’s mental model, which made the application efficient to use. In addition to the user insight and verbal feedback, both tasks took significantly less time on the PantryPal application than on our competitor’s website, which reinforces the conclusion that the PantryPal application has succeeded in increasing efficiency for this type of application. Users did, however, like the visual design on the competitor’s application, which leads us to recommend more development on the visual design aspects of the application in the future.
The most significant Caveat to this sprint was participant selection. The PantryPal team was heavily limited by the pool of participants, as well as the way that participants were selected for the Pilot Test. The UX team is also inexperienced in administering Pilot Tests, which contributed to how smoothly the study went.