What is Benchmarking?
Benchmarking is a method that allows you to compare a standard set of metrics between two ultimate states of your product in order to gauge its relative performance. The two states compared are usually an earlier version of your product and the current version of your product.
Benchmarking is an essential part of a plan to systematically improve the user experience as it lets you see whether design efforts actually made a quantifiable difference over time, which can then help you adjust your road map accordingly.
Benchmarking is also a great method for proving the return on investment (ROI) of your UX efforts. Once you have a measurable baseline of your existing product, any new work after that will make those numbers go either up or down, then you can combine that with how much effort it took you to achieve those numbers and put that all together in a cost/benefit analysis that can prove the ROI of your UX efforts.
Unlike other methods of research, benchmarking is a form of Summative User Research, which describes how a complete design performs, opposed to Formative User Research, which informs how the design will evolve during the design process.
Traditionally, benchmarking is often used to compare high-level business goals (like revenue, acquisition rates, etc ..), but it can also be used to measure the user experience of a product, which is what this method will be tailored for.
What do you need for Benchmarking?
TIME
MATERIALS
How do you Benchmark products?
Step 1: Choose what metrics to measure
In order to start benchmarking some metrics, you need to lock down what you want to benchmark first.
To do this, you may need to ask a few questions, such as:
- What do your stakeholders value?
- What does your team value?
- What will be impacted with the next set of changes you make to the product?
- What can you indirectly or directly connect to revenue or KPIs?
Answering these questions should give you a solid list of metrics to start with, but if you need more, I recommend looking at Google’s HEART Framework as well. It contains a lot of great UX metrics that you can use for benchmarking your product’s UX.
Some examples of metrics you can measure are:
- Specific feature usage metrics
- Average number of sessions per user
- Task completion rate
- Time on task
- Average number of errors per user
- And many more…
Step 2: Decide how to measure them
So now that you know what you want to measure, you have to decide on the right method to measure it all. There are multiple UX Methods to help with benchmarking, so you have to find the ones that make sense for you and your project. To do this, you can look at which methods will fit within your schedule, your skillset, the tools available to you, and your budget, then make an educated decision on what will work best.
Some common methods for benchmarking are:
- Quantitative Usability testing – This method will help you collect metrics that describe the user’s experience and performance within specific, controlled tasks. You can run the same tasks on your current product, then the new version of it to compare the differences in the results. Quantitative Usability Tests can be time consuming and hard to get a good sample size, but they can also be combined with surveys to collect both quantitative and qualitative metrics for comparison.
- Analytics data – This method is all about collecting raw quantitative data on how your product is being used by your users. If set up properly, you can quickly track almost everything that happens within your product, then look at that data again after you have made some changes to your product. This can be a cheap way to benchmark, but it will not give you much context.
- Hybrid Analytics data – If you’re looking for more context behind some of your analytics data, there are many tools out there that allow you to watch recorded video sessions of your users interacting with your product. Most of these tools allow you to filter down based on standard analytics data, and the video recordings can provide you with additional context. For example, you can filter down to find out how many people are triggering errors in your checkout flow, then watch the videos to find out more details on what they are doing to trigger that error. This is a good method for precision benchmarking, since you can then separate the different ways users can trigger that error, benchmark them, and measure your changes against those benchmarks later.
- Quantitative Surveys – For Quantitative Surveys, users are all asked the same, close-ended questions about their behaviour, background, or opinions. The data can then be benchmarked and compared against the data from the next time you run that same Survey. Surveys are a great method for benchmarking because they allow you to get responses from a large sample size with less effort than full out Usability Tests, but they are harder to get context and can fall victim to Response or Recency Bias.
Step 3: Collect your first measurement
Now that you have decided on the method you will use to benchmark your product’s user experience, you can collect your first measurement. Go through your chosen method and document the results. Now you have a benchmark, but you’re only halfway there!
Step 4: Make some changes to your product
Now that you have your benchmark data, you need to make some changes to your product so you can take a second measurement and compare it to that benchmark. For the most accurate results, make controlled changes and try not to make too many changes at once, otherwise it may be tough to correlate your changes to specific metrics.
Step 5: Collect your second measurement
Once you have made some changes to your product, you can collect another round of data using the same method you used for your benchmark. Different methods may require different amounts of time before you can collect your second round of data, so keep that in mind. For example, Usability Tests can be done immediately, but Analytics data may need a few weeks to populate enough measurable data.
Step 6: Compare and calculate ROI
At this point you have your original benchmarked data, and the new data after your changes have been made. Now you should be able to easily see the difference between the two and can understand how the changes you made are affecting your product’s user experience.
To take it one step further, you can take the difference in the metrics and factor in the cost/effort it took for you to achieve that, and then you can determine the ROI of your UX efforts. If you’re a User Experience Designer having trouble getting buy-in for UX, this is probably the greatest benefit of benchmarking.
Tips for a great Benchmark
