useit.com  Alertbox  April 2011 Users' Search Skills  

Jakob Nielsen's Alertbox, April 11, 2011:

Incompetent Research Skills Curb Users' Problem Solving

Summary: 

Users increasingly rely on individual pages listed by search engines instead of finding better ways to tackle problems.

Although some analysts questioned the finding of search dominance, it's a user behavior that gets stronger every year. Today, many users are so reliant on search that it's undermining their problem-solving abilities. Ironically, the better search gets, the more dangerous it gets as people increasingly assume that whatever the search engine coughs up must be the answer.

One recent study participant referred to "my old friend Google" as the place to go when given a task =97 a remarkable indication of how closely people are tied to search these days.

During our user testing in Asia-Pacific last month, I watched users conduct more than 100 searches for a broad range of tasks. Only once did I see a user change strategy.

Given the rarity of strategy shifts, we'd need much more data to precisely estimate how often they happen. In this round of user research, our goal was to update the Fundamental Guidelines for Web Usability seminar, so we focused on website design, not on search engine statistics.

Still, the rough estimate from our available data is obvious: users change search strategy only 1% of the time; 99% of the time they plod along a single unwavering path. Whether the true number is 2% or 0.5%, the big-picture conclusion is the same: users have extraordinarily inadequate research skills when it comes to solving problems on the Web.

In our study, for example, an interior decorator indiscriminately entered queries into any text box that caught her eye, with no understanding of which search system she was using or whether it was searching the entire Web or only the site she was on.

This example offers a striking case of confused mental models. It also highlights a big problem with search today: it doesn't facilitate any conceptual knowledge because it relies on quick in-out dips into websites.

Changing Research Strategy: An Example

Our recent research yielded only a single positive case study. In it, our test participant was trying to discover whether a friend had a cold or influenza given various symptoms (such as a sore throat).

At first, the user searched for symptoms, describing them in various ways. These simple query reformulations don't count as a strategy change because they were essentially variants on a single approach. (Without watching hours of video, I can't say exactly how many users in our study changed the phrasing of their general queries, but it was fairly common =97 about 10-20% of the time.)

Searching for symptoms was a singularly unfruitful approach. Our user was swamped with a progression of quack sites describing various superstitions and homemade remedies. Most of these were quite well-meaning discussion groups and patient support sites, but the content definitely didn't represent current medical science or trustworthy advice.

It's sad to think of the vast number of patients who get misleading medical guidance from the Internet because the main search engines currently prioritize popular sites instead of useful ones.

After a while, the user realized he was getting nowhere by searching for symptoms. He thus reversed his research strategy and started searching for the diseases, hoping to identify and differentiate symptoms between the two. This was much more successful; he found several reputable medical sites with decent symptom descriptions.

Advanced Search: Not Used

Another test participant was a lawyer who was preparing a presentation about the implications of a controversial court decision that had been handed down a few months prior to the study session. His goal was to find out what other experts had said about the decision.

Searching for various keywords that described the case, the lawyer easily found many sites with pertinent information, including news media coverage, blog discussions, and whitepapers from other law firms. However, almost all of these were written when the ruling came out and contained no analysis of the decision's long-term repercussions. They basically stated the decision and why it was good and/or bad.

Although most users never go beyond the first search engine results page (SERP), our poor user waded through many pages of SERP listings, demonstrating a valiant desire to find newer coverage.

Considering that his main criterion was recency, our user could have chosen a much easier approach: using an advanced search to filter the results by date. However, he never did so. (Remember, this was a lawyer =97 a highly educated person who regularly manages large amounts of information. Average users would have been in an even worse situation.)

In general, we almost never see people use advanced search. And when they do, they typically use it wrong =97 partly because they use it so rarely that they never really learn how it works.

The lessons are clear:

Don't assume that advanced search will help your website; you might build such features, but people will use them only in exceptional cases.

Spend the vast majority of your resources on improving regular search (simple search).

One-Track Research Strategy: An Example

The sidebar details an example of a user expending substantial effort with little result because she didn't modify her research strategy. The user racked up 22 pageviews across 8 different sites (including the search engine) trying to find the most populous city in the world. She did find an answer, but decided on it for the wrong reasons and without understanding the underlying problem =97 that there are two ways of counting city populations: with and without suburbs.

This outcome is all the more striking because the user was a schoolteacher who emphasized the need to teach students to critically evaluate online information sources.

Some users simply take the first answer they stumble across and run with it. But more careful users =97 like the teacher in this example =97 usually end up spending more time without much more benefit because they're limited by the search engine results.

After finding several widely diverging estimates of "biggest city" (ranging from 12 million to 34 million people), it would have been reasonable to change the research strategy and try to find an authoritative site on the topic of urban populations. Such a shift would likely have provided more insight than relying on the simplified lists posted on many sites that specialize in other topics and don't explain how they derive their data.

Search Is Too Good

The problem in the above examples =97 and for many other users in our recent tests =97 is that search engines are turning into "answer engines." Users are being trained to limit themselves to pages included in the SERP listing. Indeed, for many problems, the actual answer is right there. But the concept that you might have to sometimes go beyond search listings is getting lost.

For many problems, there are better approaches than simply scrolling to the bottom of the SERP. You might, for example, try to locate a site that specializes in the problem. Or =97 as in our cold/flu symptom problem =97 you might simply change the way you think about the problem.

Sadly, when one approach is so easy (and works much of the time), users never develop the research skills needed to try or even consider other approaches.

What can we do about this?

For today's Web design projects, we must design for the way the world is, not the way we wish it were. This means accepting search dominance, and trying to help users with poor research skills. For example, sites listing city populations could state explicitly that there are two ways to estimate population, rather than simply offer a single estimate without further explanation. And proper medical sites could design pages for how patients search for information, rather than for how doctors think about it.

In the long term, we should try to improve the world rather than design to suit its shortcomings. One example of how we might do this is to teach better Internet research skills in schools.

Learn More

Mental models and the design lessons from psychology will be discussed in the full-day seminar on The Human Mind and Usability at the annual Usability Week conference.

The conference also has a 2-day seminar on Writing for the Web. The seminar on Web Page Design discusses how you can help users who arrive through deep links instead of through your homepage.

> Other Alertbox columns (complete list) 

> Sign up for newsletter that will notify you of new Alertboxes

Copyright =A9 2011 by Jakob Nielsen. ISSN 1548-5552