UX testing methods: when to use what (and why)

Five users find 85% of usability problems. That's been the gospel of UX testing since Jakob Nielsen published the finding in 2000. But that famous study doesn't account for the six weeks it takes most teams to actually recruit, schedule, and test those five people. By the time you've watched the last recording and synthesized your notes, the feature has already shipped to production.

UX testing is the practice of evaluating a product's interface by observing how people, or AI users that simulate people, interact with it to identify where they struggle, succeed, or give up. It's the single most reliable way to find problems before your users do, and it comes in more flavors than most product teams realize.

The real challenge isn't understanding the methods. It's picking the right one for the question you're trying to answer, with the time and budget you actually have.

#What are the main types of UX testing?

UX testing methods split into two broad categories. There are methods where you watch people use your product, and methods where you inspect the product yourself.

The "watch people" category includes moderated testing, unmoderated testing, guerrilla testing, and AI-driven testing. The "inspect it yourself" category includes heuristic evaluation and cognitive walkthroughs. Then there's A/B testing, which sits in its own lane because it measures preferences across a population rather than observing individual behavior.

Here's how they compare at a glance:

#When should you use moderated usability testing?

Use moderated testing when you need to understand the "why" behind user behavior, not just what happened. A trained moderator can follow up on confusion in real time, probe emotional reactions, and catch the subtle moments that recordings miss.

Moderated studies shine in three situations. First, when you're testing a complex, multi-step flow like onboarding or checkout where context builds across screens. Second, when your target users are specialized (think medical professionals or financial advisors) and you need to understand domain-specific mental models. Third, when you're early in the design process and your prototype is rough enough that participants need guidance to get through it.

The tradeoff is real, though. MeasuringU's cost analysis puts a moderated study with 5 participants anywhere from $415 to $1,680 just for recruitment and incentives, and that's before you factor in the moderator's time, note-taking, and analysis. For 20 participants, recruitment alone runs $12,000-$15,000.

#When does unmoderated testing make more sense?

Unmoderated testing wins when you have a clear, specific question and need answers from a larger group of people without the scheduling overhead. Participants complete tasks on their own time, in their own environment, which means you can run a 50-person study in the time it takes to schedule three moderated sessions.

This method works best when you're validating whether a specific flow works, not exploring open-ended questions about user needs. If you want to know whether people can find the pricing page from the homepage, unmoderated testing gives you that answer fast. If you want to know why they're confused about your pricing model, you need a moderator in the room.

The cost savings are significant. An unmoderated study with 5 participants costs roughly $250 on the low end, and platforms like Maze or Lyssna can get responses back within hours. The catch is that you lose the ability to ask follow-up questions, so your task design and instructions need to be tight.

#Is A/B testing a substitute for usability testing?

No, and confusing the two is one of the most common mistakes product teams make. A/B testing tells you which version performs better. Usability testing tells you why users struggle. They answer fundamentally different questions.

A/B testing is great after you've done usability work. You've identified the problem, designed two potential fixes, and now you need data on which one moves the needle. Running an A/B test without usability context is like optimizing a recipe without tasting the food first. You might land on a version that converts 3% better, but you'll never know you could have converted 30% better if you'd understood the underlying problem.

The teams that get the most out of A/B testing are the ones using usability insights to generate their hypotheses. "Users are confused by our pricing toggle" is a usability finding. "Let's test a dropdown vs. a toggle and see which converts better" is the A/B test that follows.

#What about heuristic evaluation?

Heuristic evaluation is the UX equivalent of a code review. Instead of watching users, you have experts walk through the interface and flag violations of established usability principles, typically Nielsen's 10 heuristics.

It's fast, relatively cheap, and catches the obvious stuff. Three to five evaluators working independently will surface a solid list of issues in a day or two. The method is particularly useful when you're on a tight budget, when you want to clean up known problems before spending money on user testing, or when you need to evaluate a competitor's interface.

The limitation is that experts aren't users. They'll catch "this error message doesn't tell the user what to do next" but they'll miss problems that only surface through real behavior. Every team that's run both a heuristic review and a usability test knows the feeling of watching a real user struggle with something three experts rated as "fine." That gap between expert prediction and actual user behavior is exactly why heuristic evaluation works best as a complement to testing, not a replacement.

#Does guerrilla testing actually work?

Guerrilla testing, sometimes called hallway testing, means grabbing people in a coffee shop or office hallway and asking them to try your product for 10-15 minutes. It sounds informal because it is, and that's both its strength and its limit.

For early-stage concepts and quick directional feedback, guerrilla testing is hard to beat. You can run a session over lunch and have usable insights by the afternoon. It costs almost nothing, forces you to keep your research questions focused, and builds a habit of testing regularly rather than treating it as a quarterly event.

The problem is participant quality. The person at the coffee shop is almost certainly not your target user. If you're building enterprise security software, a random stranger's feedback on your dashboard tells you very little. Guerrilla testing works for universal usability questions ("Can someone figure out how to sign up?") but falls apart for anything that requires domain expertise or specific user context.

#How does AI-driven testing fit into the mix?

This is where the landscape has shifted dramatically in 2025 and 2026. AI-driven UX testing uses AI users with realistic personas, complete with patience levels, tech literacy, attention spans, and specific goals, to navigate your product the way real people would.

The value proposition is straightforward. You get the behavioral richness of a moderated study at the speed and scale of automated testing. At Flawd, we've built this approach around personas that actually make mistakes, get impatient, and give up when things are confusing, because that's what real users do.

Here's what AI-driven testing is genuinely good at:

• Broad coverage testing. Run 20 different personas through your checkout flow in an afternoon, each with different goals and behavioral constraints. A moderated study at that scale would take months.
• Regression testing for UX. Every deploy, every feature flag, every copy change can be tested against the same personas to catch regressions before users hit them.
• Finding problems nobody thought to look for. Because AI users aren't following a script, they stumble into issues that scripted test plans miss entirely.

What makes AI-driven testing particularly powerful is that it combines qualitative depth with quantitative scale. You get session recordings that show exactly where each persona hesitated or gave up, plus aggregate patterns across dozens of sessions that reveal systemic issues. It's the kind of insight that used to require weeks of moderated sessions and hours of synthesis — delivered in an afternoon.

#How do you choose the right method?

Start with your question, not the method. Every UX testing decision comes down to three things:

1. What are you trying to learn? "Why are users dropping off at step three?" needs qualitative depth (moderated testing or AI-driven testing with detailed personas). "Does version A or B convert better?" needs A/B testing. "Are there obvious usability violations?" needs a heuristic review.

2. What stage is your product in? Early concepts benefit from guerrilla testing and heuristic reviews because they're fast and cheap. Mature features benefit from unmoderated testing for benchmarking and A/B testing for optimization. AI-driven testing fits across stages because you can adjust the depth and breadth of your persona configurations.

3. What's your timeline and budget? If you need answers this week, guerrilla testing, heuristic evaluation, and AI-driven testing are your options. If you can wait a month, moderated testing gives you the richest data. If you're running continuous discovery, a mix of unmoderated testing and AI-driven testing keeps insights flowing without burning out your research team.

The most effective product teams don't pick one method and stick with it. They layer methods based on what they need to learn at each stage. A common pattern that works well is running AI-driven testing through Flawd for broad coverage and regression testing, using the findings to design focused moderated studies for the biggest open questions, then validating final designs with A/B tests.

#The method-stacking approach that actually works

Here's a practical framework for combining methods across a product development cycle:

• Discovery phase. Heuristic evaluation to audit the current experience, plus guerrilla testing on early concepts. Low cost, fast turnaround, enough signal to set direction.
• Design phase. AI-driven testing on prototypes and staging environments to catch issues across multiple persona types. Moderated testing for the two or three flows where you need deep qualitative understanding.
• Pre-launch. Unmoderated testing at scale to benchmark task completion rates. AI-driven testing with diverse personas to stress-test edge cases.
• Post-launch. A/B testing for optimization decisions. Continuous AI-driven testing for regression monitoring. Periodic moderated studies to stay connected to real user experiences.

The $1-to-$10-to-$100 rule still holds. Fixing a usability issue in design costs a tenth of fixing it in development, and a hundredth of fixing it after launch. Every method on this list pays for itself many times over when it catches problems early.

The teams that ship the best experiences aren't the ones with the biggest research budgets. They're the ones who match the right method to the right question and test more often than feels comfortable. If you're only testing once a quarter with one method, you're leaving problems on the table for your users to find instead.