Human Blackbox

Problem
Objectively measuring the quality of a solution created by a participant can be difficult if the solution is not easily quantifiable and can only be subjectively judged.

Solution
Instead of trying to give a subjective judgment on a solution, which is open to bias (see Coding Calibration), create a follow-up experiment where new participants use the solutions from the first experiment to solve a particular task in a way that can be quantified. The participants in the follow-up essentially become blackboxes that we do not have to open, just study their outputs given specific inputs.

Consequences
This pattern requires you to add a second experiment, which is both costly and time-consuming. It also requires you to design a new task for the second experiment that uses the output from the first and yields a result that can be easily quantified (i.e., either true or false).

Examples
To our knowledge, this pattern was first used by Dwyer et al. (2009) in work that builds on an earlier study of user-generated graph layouts (vanHam and Rogowitz 2008). However, where van Ham and Rogowitz from that study attempted to qualitatively describe and quantify the user-generated layouts resulting from an experiment, Dwyer et al. instead added a second experiment where participants performed several graph tasks using the user-generated layouts. The performance of participants solving these tasks in the second experiment became robust metrics of the quality of each user-generated layout without the authors having to resort to subjective judgments.