How to avoid Bad User Research

A common belief amongst companies and teams who develop and design digital products is that every kind of user research is better than no user research at all. Of course, we are big fans of everything related to UX usability, user research and -testing. Still, we don’t believe that “some is better than none” always holds true for user research and -testing.

The research data gathered in these processes provide input and information that is used later on when the product development team makes design decisions. And that’s exactly where the problem with bad research data starts: These decisions can only be well informed if the underlying information is accurate. After all, decisions are only as good as the data that fueled them. In the end, it doesn’t really matter how good your UX designer or team is if the input fueling their decisions is inaccurate.

Reasons for low-quality research data

There is a number of problems that can arise during the research process that will lead to skewed data or wrong conclusions. Some of the most common that can arise during the research process are listed below:

Testing the wrong Users

Every researcher knows that the participants that are surveyed or tested should represent the actual user group of the final product as closely as possible. When participants don’t represent the target users of your product you’re simply capturing the wrong opinions, wishes needs and problems. Effectively you will be designing a product that fits the needs of the population you tested. But said population is not really relevant for your product.

They can be people who are unlikely to use your product later on, not qualified enough to use your product, to qualified to use your product… the list goes on. Imagine you want to test a product designed for power-gamers but your participants are people who don’t even play video games on a regular basis. In this scenario, you’re unlikely to get good input.


Even if you know the characteristics of the group you need to recruit you can still accidentally end up with the wrong participants if you use the wrong screener questions when recruiting participants. Keep in mind to not only ask for socio-demographic data but also ask behavioral questions. Consider double checking whether participants fit your criteria by asking them appropriate questions right before they begin the testing process.

Trying to solve the wrong problem

In order to help your users and improve your product you first need to really understand their problems. Only then can you start to try solving them. Understanding your users’ problems becomes harder with wrong assumptions you make regarding your users. Your research questions are usually based on some kind of previous assumptions. If those assumptions are faulty, your research will continue in the wrong direction. Not every assumption should be accepted without asking where it came from and what it is based on. Beside questioning your assumptions critically it can also help to try to think outside the box sometimes.

Drawing the wrong conclusions

Even if you asked the right participants the right questions, it’s still possible that you end up not interpreting your results correctly. This can happen because of certain kinds of cognitive bias or because you’re trying to get the most out of your data, trying really hard to interpret it and therefore rushing to wrong conclusions.


Overcoming bias can be really hard since biases are subconscious. You are way more likely to quickly accept findings that agree with opinions you already had before you started testing. Along the same line is something odd or unexpected happens and catches your attention, you might quickly consider it an interesting new finding when it could just as well be an outlier or random happening. Try to not transform surprising findings into assumptions or even recommendations without verifying them first with additional research.

Excluding important user groups

If there are multiple user groups for your product you should include representatives of all groups in your testing process. If you lack the time and/or money to test all user groups, it’s, of course, better to focus on the most important group than not to test at all.

However, this neglect of whole user groups might result in a product that isn’t usable for everyone you designed it for. When you discover a new user group you didn’t consider before you should conduct research with representatives of said group as soon as possible.

Concentrating on only User Testing or only on User Research

As stated above, user research fuels user testing. It’s hard to run good and efficient user tests with the right users if you don’t exactly know who your user group is. Also, knowledge about your users gathered by doing user research won’t be nearly as helpful for the development of your product as a test involving realistic tasks. You should avoid doing one without the other.

How to gather reliable Data

Having considered all the factors listed above you should also try to test products in the most realistic setting possible to make sure users actually use them in the same way they would outside of your research. Consider that people will still behave differently in a research setting.

Also don’t blindly trust your users’ feedback. Sometimes certain design choices might not be ideal but still the best solution considering the circumstances. Fixing a minor problem that shows up with your product might cause a way bigger problem the users you asked didn’t even consider. Always favor task-based tests over surveying users or asking their opinions. Sometimes users might not know what they want or need before they try to actually accomplish a certain task with your product.

It’s also an advantage if you have a research team and not just a single researcher. That way findings can be discussed to assure that you get the best possible research quality. Keeping insights contained within a limited number of people or only communicating it through them is not ideal as they will subconsciously filter what they pass on. The whole team should take the time–at least every now and then–to observe users, recorded testing sessions or other raw data such as audio or text files.

That way you can make sure that your research data isn’t only used to support your assumptions and previous findings but rather to uncover new insights to shed light on areas you haven’t looked at before.

You need a tool that helps you analyze your research results? We’re working on the solution for you: Consider.ly makes it possible for you and your team to create searchable recharche findings and helps you spot patterns assisted by AI. For any questions, please send us an E-Mail.