Investigating the Effectiveness of Craigslist Search Functions
Our team designed and conducted a usability test to assess Craigslist features most likely to affect user satisfaction and site abandonment rates.
Summary of study
Craigslist is an online marketplace that functions as a platform for community members and local dealers to buy and sell goods from each other. The purpose of this study is to assess the usability of Craigslist’s search filter and email notification functions. After observing participant behaviors and collecting their feedback as they performed tasks using these features, we analyzed the data and put forth major findings and recommendations for improving user satisfaction and reducing site abandonment in regard to these features.
According to participants, site navigation was ambiguous at times, which caused confusion among participants and slowed their browsing. In addition, participants had difficulties finding and using “save search,” option to set up email alerts, and several aspects of the email notification setup process were difficult to find and understand, particularly because of a lack of clear and appealing visual cues. Participants also wanted to see more filtering and tagging options, which were comprehensive for some searches but lacking for others.
​
Participants
We recruited eight participants in total. All of the participants were required to be at least eighteen years old, and have at least basic experience navigating the web. In this way we were able to investigate not only user satisfaction, but also the discoverability and learnability of certain Craigslist functions by comparing the behaviors of experienced users with that of inexperienced users.
8
Total participants
Age
​
Showed interest in buying used goods
8
18+
?
Experts
Non-experts
Has used Craigslist 5 or more times this year or has a Craigslist account
Has used Craigslist fewer than 5 times this year and does not have a Craigslist account
Methods
Having each conducted a heuristic evaluation and cognitive walk through of the Craigslist website, our team was able to generate hypotheses about the portions of the site that would be most problematic for users. We designed three tasks involving Craigslist features that were most likely to impact user satisfaction and site abandonment rates.
​
Test Structure
In order to ensure uniformity among the tests, participants were read a pre-made script. Each of the usability tests were structured identically. Each test session was approximately one hour, with 15 minutes total allotted to the combined following sections: introductions and overview, pre-test questions, post-task questions, and post-test questions. The remaining 45 minutes were given to the completion of the tasks. Participants were allowed to take as long as they needed to finish the tasks.
3x
General questions
Task
Post task questions
Post test questions
Tasks
Participants were asked to complete three major tasks, and they will be provided the following scenarios to help them understand the context and complete the tasks:
​
-
A relative has left you $7,000 to spend on a car. Use Craigslist to find a vehicle you might be interested in.(after participant has selected car) Your current job requires you to make long trips regularly so you want a car without too many miles on it. Find a car that has fewer than 140,000 miles on it.
-
You recently saw shelves like these at your friend’s house and you think you might like something similar for storage in your own room. You take to Craigslist to see if there are any second hand shelves like them available. You frequently get home from work in the evening and don’t have time to drive more than 15 minutes to go buy them.
-
You want new headphones but you’re not willing to spend more than $40. You look on Craigslist for headphones you might like that are within your price range. You are not sure if these are the best headphones you can get for that price. Set up email notifications to alert you when new headphone listings are posted in a similar price range.
​
​
Data Logging
During the usability tests we collected video of participant's face and their screen activity, as well as audio. This allows for “think aloud” data to be recorded in coordination with timestamped notes. Each task was categorized as either a pass or fail. The completion condition of each task for participants is to find the item corresponding to our requirement, or to set up the alert successfully. Additionally, the number of times the participant refreshes/updates the search, and number of “errors” (things that the participant had to correct, or that had to be redone) will be recorded as they occur.
Metrics and Evaluation
The data collected from the usability tests was used to identify where errors were most common, as well as the average amount of errors per page. Think aloud data and post-task questions were analyzed for common themes and insights.
​
After completing all eight usability tests, our research team exported all videos and arranged all handwritten and typed notes to be used for affinity diagramming. After going through all the notes, the team put important
findings on individual sticky notes and categorized them. After completing the affinity diagram, the team had a general understanding about the difficulties the participants had met. The list of findings was put into an Excel spreadsheet. The team then looked over the videos and notes again, highlighting relevant evidence for each finding. Findings were compiled into a preliminary report, and then later into a complete usability study document.
Listing participant answers and data in a spreadsheet
Affinity diagramming process for data organization
Key findings
"Save search button is not in a place I would have expected it to be. There's nothing up in that corner. All the rest of the options are on the left."
-P2
to
Three findings presented themselves as imminently actionable and directly related to Craigslist's user satisfaction and site abandonment rates.
​
Ambiguous Site Navigation
Craiglist lacks a clear “main track” that will direct the majority of users where they want to go. Craigslist provides many paths to navigate the site, but some users spent an undue amount of time deciding how to navigate to a desired page. By clarifying a distinct path to navigate the site, Craigslist can likely reduce the “overwhelming” impression of the site and direct users more efficiently and satisfactorily to their goals.
​
​
​
​
​
Save Search
Second hand goods cannot be available all the time, and it is important to support a continued shopping experience even if the goods are not presently available. For this reason, the email notification system is vital for keeping users engaged with the site even when they do not immediately find what they are looking for. The “Save Search” function allows participants to set up email notifications, but in every single test the participants struggled to find this button on the search page. The label “Save Search” was not clear, the location of this button on the search page did not meet user expectations, and, once found, the checkbox interface to turn notification on confused participants. By resolving these usability issues Cragislist can keep users connected to their second hand shopping experience even when their target item is not presently available.
​
​
​
​
​
​
​
​
​
​
​
​
​
Filtering and Tagging
The post task feedback showed that participants appreciated the availability of search filters to find their desired vehicle. The filters available in the Cars & Trucks search category are plentiful, including everything from make and model to odometer information. Some participants expressed a desire for more filter options on items other than vehicles. The third task of finding floating wall shelves, for example, might have been easier had there been a more shelf-specific means of sorting search results. During Task 2, all of the participants spent a significant amount of time revising their search terms in the keyword search. This process could be greatly accelerated by creating more filter options for items other than vehicles, ultimately leading to a more satisfying shopping experience.
​
"Save search is not necessarily what I would think of for setting up email notifications.”
-P5
Reflection
If we were to conduct this study again, there are several things that our team would do differently. Some of the flaws in our team organization caused us to forego an actual “pilot.” We ended up scheduling 3 participants our first day (one of which later had to reschedule.) However, after completing 2 of the usability tests, it became clear that changes were needed. The Test Kit materials were revised to be more suitable for the study: namely, the moderator was given the data logging sheet, and the introduction and tasks in particular were reworded to be more clear. However, these changes were not implemented until after 3 participants had already been tested. If we had made use of a proper pilot, our tests would likely have run more smoothly, and we would not have to do as much work later collecting our data.
In addition to the points listed above, if we were able to conduct this study again, our team agreed that a wider pool of participants would have been beneficial. Not only would it have given us more data to work with, it would have helped to eliminate bias. One notable concern was that, after we had run the study, we realized that our expert population was entirely male, while our non-expert population was entirely female, purely by coincidence. A wider pool of participants would probably have solved this issue.
TIMEFRAME
September - December 2017
​
TEAM
Valerie Huang, Jackie Yu, Chenying Yuan
​
MENTORS
Liz Sanocki, Bonnie Chinh
​
TOOLS
UX Cloud, Zoom, Microsoft Office