Complementary Studies

Problem
In designing a visualization user study, we are often faced with a choice of a rigorous and unrealistic study, or a realistic but ad hoc one. Achieving both in the same study is often impossible: for a rigorous study, we need to be able to generate balanced trials, which means that the data cannot be truly real. For realistic data, on the other hand, we run into learning effects, variability in the trials, and difficulty controlling all aspects of the task and dataset. In other words, the rigorous toy study lacks ecological validity (conformance to a realistic situation), whereas the ad hoc study lacks internal validity (confidence of the measured results actually being a function of the experimental factors).

Solution
The obvious solution for remedying the above problem is to include both kinds of studies in your paper, and have them complement each other. The rigorous toy study will probably be the backbone of proving that your system or technique actually works in the general (but unrealistic) case. The realistic ad hoc study, on the other hand, will serve as a much-needed sanity check and help to convince the reader that the work is applicable to the real world.

Consequences
Using the Complementary Studies patterns essentially requires twice the resources in time, money, and preparation of conducting just one of the two possible studies. Beyond that, describing the details of two studies in the same research paper may be costly in terms of space.

Examples
In evaluating visual search performance for the Color Lens technique (Elmqvist et al. 2011), which dynamically adapts a color scale to fit the range of data values within a magic lens, there was an option between searching for a variable-strength feature (a circle) in a random noise background, and a named feature in a real photograph (i.e., "find the deer in this picture of a forest"). Instead of choosing just one option, both studies were conducted and reported on.