Know-how of Card Sorting

Know-how of Card Sorting

2018｜Type: Original｜Tag: Theory

What’s card sorting?

Card sorting is a simple and easy method to organize information in an understandable way to help design or evaluate information architecture. Usually, it’s conducted at an early stage in user research. The researchers encouraged participants to conduct card sorting according to their preferences by using several topic cards labeled with words and pictures. Card sorting can help researchers gain a lot of insights, and even have a chance to perform interviews to know the reasons and values of their results.

Methods in Card Sorting

Open Card Sorting: Given the topic cards to be sorted. Participants were asked to organize topic cards into groups based on their understanding and named the groups the way they described their content.
Closed Card Sorting: Given the topic cards are to be sorted and grouped. Participants are asked to organize topic cards into the given groups based on their understanding.

Open card sorting is designed to understand how users create and name groups and closed card sorting is designed to verify the information architecture and naming. Generally, after conducting an open card sorting, one or more closed card sorting can be conducted if you need to verify whether the information architecture is appropriate.

Technique in card sorting

One-to-one sessions: This requires in-person sessions with an observer, a participant conducts independently, which gives a deeper understanding of the participant's mind, but it has a higher time cost.
One-to-many sessions: This requires concurrent sessions with an observer, every participant sorts a set of cards independently, which could be fast to collect, but also more materials to be prepared and less understanding of every participant.
Team sessions: This requires participants to sort a set of cards as a group. Collaboration can promote card sorting faster, but group atmosphere, leadership as well and relationships between group members need to be taken into account.
Remote sessions: There is no need to have usability members, participants do it individually. This can be conducted across regions, but the process of conducting is mysterious.

Why use card sorting?

A user is conducting card sorting

Inviting users to participate in card sorting helps to understand the information architecture from the user's perspective. Understanding how users organize can help product teams verify that classifications are in line with user expectations and further optimize the information architecture.

How to conduct card sorting?

Create topic cards

Prepare topic cards or picture cards according to the topic you want to test. Each card has only one topic. 30 to 40 is preferable, and 50 to 60 is also possible. Excessive amounts can easily cause participant fatigue.
Prepare several blank cards for participants to add topics themselves.
Consider using different color cards for participants to name the group.
Consider taking marks where it is hidden so that researchers can analyze it conveniently.

Prepare

Invite users. Research has shown that the correlation coefficient between the 15 sample sizes and all users reached 0.90. Report
Estimate time. Providing estimated time helps participants build expectations.
Enough space. This allows participants to tile the topic cards on the table or paste it on the wall.
Record at any time. Participants’ thoughts, reasons, and frustration can be recorded by usability members.
Offer rewards. Several prizes can be prepared for participants.

Conduct

Clarify the purpose. Briefly explain the purpose of card sorting to the participants. In the open card sorting, indicate that participants are required to name their custom groups; in the closed card sorting, indicate that we want to know how they understand their definition of these groups.
Offer random cards. Random cards appear to help participants sort carefully.
Observe. Try not to interrupt the participants. Allow participants to add topics with blank cards, or discard useless topic cards.
Think loudly. Encourage participants to think loudly and help usability members keep track of it.
Offer rewards. Several prizes can be provided for participants.

Analyze

Records timely. Take photos of topic cards that have been sorted, and take note of group names and topic cards of the group. Shuffle cards for the next participant.
Qualitative information. Perform qualitative analysis based on user reviews.
Quantitative information. Quantitative analysis of grouped information using hierarchical clustering or multidimensional scaling.

Also, you could conduct card sorting online!

There are many desktop tools and online tools that can run card sorting, and most tools have basic analysis features. Common tools are as follows:
Cart Sort (Windows)
xSort (Mac)
OptimalSort (https://www.optimalworkshop.com/optimalsort)
UsabilityTest Card Sorting (https://www.usabilitest.com/card-sorting)

How to analyze opening card sorting

Theory

Study the similarity between all cards by measuring the distance between each pair of cards. Therefore, any standard statistical method for studying the distance matrix can be used, such as hierarchical cluster analysis and multidimensional scaling.

In mathematics, a distance matrix is a square matrix (two-dimensional array) containing the distances, taken pairwise, between the elements of a set. —— Wikipedia

Tools

Donna Spencer’s card sort analysis spreadsheets

Donna‘s template provides co-occurrence matrix spreadsheets which come in two versions, 20×200, and 40×400.

SPSS Statistics

Many commercial statistical tools could be performed, including the Unistat plug-in for Excel on Windows. Take a look at the hierarchical cluster analysis of SPSS as follows.

Path: SPSS - Analyze - Classify - Hierarchical Cluster

Steps

1. Turn single participant data into an N * N symmetric matrix.

According to different definitions, there are two kinds of pre-processing paths when transferring data, but they all work the same way:

The distance within the group is 0 and the distance outside the group is 1, which is the dissimilarity matrix.
The distance within the group is 1 and the distance outside the group is 0, which is the co-occurrence matrix.

Take the co-occurrence matrix as an example, set the distance between the cards in the same group as 1, and others as 0, take records, and then we get a matrix which is filled only be 1 or 0.

Matrix Transformation

2. Add matrices of all the participants

We will have a co-occurrence matrix after matrices addition. Due to the need for analysis data, matrices shouldn’t be left blank, the diagonals are treated as 0.

The diagonals are treated as 0 in the dissimilarity matrix.
The diagonals are treated as the same as the number of users in the co-occurrence matrix.

Taking the co-occurrence matrix as an example, the symmetric matrix for all participants is as follows.

Matrix Addtion

3. Adjust data format

In the variable view, adjust the variable name to a fixed-type and the variable frequency to a fixed-distance.

Adjudst the Variable In the Variable View

4. Hierarchical cluster analysis

When using SPSS to run hierarchical cluster analysis, different cluster methods and measurement intervals for calculations can be used. Between-Groups Linkage, Within-Groups Linkage, and Ward’s Method are more effective cluster methods based on our experience. Also, since we are clustering the cases, we use the Euclidean distance or Squared Euclidean distance of the Q-type clustering on the measurement interval.

Cluster Method

Between-Groups Linkage
Within-Groups Linkage
Nearest Neighbor
Furthest Neighbor
Centroid Clustering
Median Clustering
Ward’s Method

Measures for Interval Data

Euclidean distance
Squared Euclidean distance
Cosine
Pearson correlation
Chebychev
Block
Minkowski
Customized

Results

View classification results from Vertical Icicle and Dendrogram.

1. Vertical Icicle

Observe the vertical axis and cut horizontally by adding cut lines.
Observe the 0 to cut line interval, the interval is the group.

Vertical Icicle

2. Dendrogram

From left to right, you will see which cards are clustered together at the beginning;
From right to left, the number of groups can be understood by adding cutting lines, and the number of groups can be set as either the average value or the target value.

Dendrogram

Typically, an opening card sort can be immediately followed by one or more closed card sorting, through which the appropriateness of the information categorization is verified.

How to analyze closed card sorting

Theory

Define the measure as the proportion of participants placing theme cards in each group. A large difference in the proportion of each theme card between groups is a more confident classification, while the opposite is a divergent classification.

Tool

Basic statistics can be done using Excel.

Steps

Invite participants to match theme cards to groups.
Transform each participant's data into an N * N symmetric matrix.
Add the N * N symmetric matrices of all participants.
Convert the frequencies into percentages and mark the maximum percentage for each theme card.
Find the average of the maximum percentages, which measures the validity of the group name.

When we enumerate multiple classifications and need to validate one of them, we need to invite a consistent number of different participants based on each classification. Also, need to consider:

The number of groups is consistent across multiple classifications, which can be measured directly using the average of the largest percentage.
If the number of groups is not consistent across multiple classifications, the average of the difference between the largest and second-largest percentages can be used.

Vertify

What I have learned

Marking numbers to the topic cards really helps improve data collection. From offline data to online is heavy and inefficient, combined with the numbers could be helpful.
The wording of the topic cards should be as precise as possible to avoid deviations and ambiguities. If you can't avoid it, you can explain it on it. Otherwise, incomprehensible topic cards will be sorted by users into unexplainable categories.
Last but not least, users are not necessarily right. When the subject covers a wide range of fields and there is an unavoidable deviation in the understanding of the subject card, the result of participants is limited by knowledge, therefore an expert method can be considered to conduct closed card sorting.

Reference

https://www.usability.gov/how-to-and-tools/methods/card-sorting.html
http://www.designkit.org/methods/24
https://zh.wikipedia.org/wiki/%E8%B7%9D%E7%A6%BB%E7%9F%A9%E9%98%B5
http://blog.sina.com.cn/s/blog_777d52410101ilyz.html