When Jared Spool's Web Site Usability: A Designer's Guide was published, I was so interested that I packed it as carry-on reading for my vacation. I had found very few published accounts of empirical studies of web usability issues, and was anxious to compare notes with a fellow professional's web research methods and findings. The book is a quick read, so by the time the plane landed, I was thoroughly disappointed. Mr. Spool sets high expectations in the early pages with these three humble claims: that his book presents "groundbreaking research on web site usability;" that this research "provides actual data - not opinions - about what makes web sites usable;" and that his results are "really cool scientific findings." But the book fails on all three counts.
Because the authors neglect to describe their research methods up front, the first errors the reader will notice are problems of logic, definition of terms, and overgeneralizations based on limited observations. The first chapter presents a set of "major implications," each of which is meant to debunk some common-sense idea of web design; for example, Implication 1 is "Graphic Design Neither Helps Nor Hurts." However, the reader who tries to follow the logic behind the titillating assertions will find it mortally flawed. In this example, a little digging reveals a misunderstanding of the meaning of graphic design, which the authors interpret as the quantity of picture elements in a given web page or site. This misinterpretation leads them to conclude that if a site with few pictures (described as a "nearly 'design-free zone'") fares better with subjects than do other sites with many pictures, it is because graphic design is unimportant.
Some of the conclusions are directly contradicted by reported results. The authors conclude, for example, that "The more white-space there was on a site, the less successful users were at finding information." Yet Edmund's, which uses white space very effectively for visually separating the various informational categories, was ranked best for ease of finding information.
The research methodology is mentioned only very briefly, toward the end of the book. Even then, the little information offered is enough to raise serious questions about what is not revealed. Here is a sampling of the facts I could glean: The researchers did not consider the sites' intended audiences when selecting subjects to evaluate them. The sites examined were aimed at vastly differing audiences ranging from kids (Disney) to durable-goods comparison shoppers (Edmund's) to small business owners (Inc.). Yet a single group of subjects was chosen to represent all the sites' users in the testing.
Test tasks did not necessarily resemble likely end-user tasks, and the purposes of the sites were disregarded. Obviously, the effectiveness of a site should be evaluated in the context of the reasons for the site's existence. One site may be designed to facilitate the users' speedy navigation to information the user is seeking out, while another may intentionally divert users to certain pages to attempt to sell impulse items. It is not meaningful to compare these two types of sites on the same criteria.
The test data are a sloppy combination of between- and within-subject ratings. The authors explain that "...each [subject] tested as many web sites as possible in [the three-hour time allotment] (no [subject] tested all the sites)." The ratings tables do not include the number of observations used to calculate each "average" rating score. No variables were held constant across sites. Therefore, the reasons asserted for any differences between sites' ratings are strictly conjecture on the part of the researchers. Even more distressing than methodology described are the questions left unanswered. These include some as basic as: How many subjects participated in the testing? What incentives were used to motivate the subjects' participation? What were the demographics of the subjects?
The most valuable piece of information in this book is the one uncharacteristically candid remark tucked away in the Foreword: "...no one should accept our reasoning without question." Subtract the two final words of this statement, and you will have a pithy summary of my review.