- Participants usually sort only a portion of the content, so the architecture they generate is valid only for the items included in the sort.
- If the participants are subject matter experts, they are unlikely to generate a structure that works for non-experts. Conversely, if they are not experts, their lack of awareness at times will make it hard for them to create a good structure. If the participants are unmoderated, they won’t be able to get answers to their content questions.
- People will vary in their understanding of the rules for the sort. For example, can content be repeated in multiple locations? Can an item be split into two items, or combined with another, to make the items fit better with the approach they have chosen? Can an item be left out if it doesn’t seem to fit? Experience shows us that even meticulously written instructions won’t always fix these problems.
It’s human nature to seek the quick answer, the simple solution, or the simple rules; to want to believe in guarantees. We want to believe that a new golf club will take several strokes off our game, or that a pill will help us lose weight without having to diet, or that an apple a day will keep the doctor away. The same is true when designing and testing. We want to believe that five users can uncover 80% of our usability issues, or that a website will be usable if everything is three clicks from the home screen, or that an open card sort will generate the information architecture for a website. Of course, none of these are true, but let’s focus on the last one. A card sort is a traditional tool for the design community to address website organization. In open card sorting, participants are provided with content on index cards and are asked to arrange them into groups and subgroups, give names to the groups, and thereby create a structure that will hold all of the content. Claims of the effectiveness of this approach on generating a final architecture vary. You can find references that claim fifteen people will show a very high correlation of .90. You can read that this task can be done on the web in an unmoderated study and that the results can be analyzed using automated cluster analysis software. Sounds simple, sounds great, sounds too good to be true. And it is too good to be true. If it were true, all information architectures, menu structures, and other things that need to be groups would be perfect. Open card sorts are for generating data, not for providing answers. Here are three truisms of an open card sort: