Paper Baseline

Problem
Determining how people make sense of data "in the wild" can often be obscured by the visualization tools themselves. The participants may not be fully fluent in using the tool, and the tools--being research prototypes--may not have an optimal interaction design. In fact, first developing a prototype tool with the intention of supporting people's "natural" mechanism of interacting with data without knowing this mechanism is actually somewhat counterproductive.

Solution
Instead of designing a visualization system to use as an evaluation platform, conduct an entirely paper-based evaluation. If appropriate, combine the paper-based study with a visualization-based study to compare the two.

Consequences
This pattern reduces the need for costly and time-consuming software development. However, it is most suitable for formative design, and will obviously not yield results for interactive behavior. Furthermore, while a computer-based study is easy to instrument, this is not the case for a paper-based one; you may have to resort to videotaping participants, or keep careful observation logs.

Examples
Kang et al. (2009) include a Paper Baseline condition in their qualitative study of Jigsaw (Stasko et al. 2008). Isenberg et al. (2008) and Robinson (2008) both base their studies of collaboration data analysis solely on paper printouts.