Pilot Study

Problem
Determining the minor details of an evaluation can be difficult. Such details include the difficulty of the trials, task formulation and design, training sessions, blocking and order, data measurement, and overall study balance.

Solution
Perform several dry runs of the experiment with unbiased participants. Each dry run (or pilot) should mimic a real experimental session as closely as possible, but you may want to make changes to the experiment after each pilot to improve its design. Pilot participants should be objective and unbiased to yield the most benefit (in other words, involving a project member as a participant is a bad idea), but having expertise in human subject experiments is helpful since it allows the participant themselves to give informed advice on how to improve the study. Determining how many pilots to run is open to debate; one or two is often too few, whereas three or more allows for achieving stability in the changes made to the study. It is also very important to conduct the planned statistical analysis out of the collected data, which often helps the experimenter to identify any errors and mistakes in the data collection.

Consequences
Running one or several Pilot Studies is a very common practice in human subjects evaluation, but deserves being highlighted here as a pattern in recognition of its prominence. Even though Pilot Studies add to the time investment of performing the evaluation, they are truly invaluable in detecting problems early, and therefore often repay themselves many times over the course of a study.

Examples
Pilot Studies are typically not highlighted in research papers, but are nevertheless used to calibrate most reported evaluations. One concrete example is the Pilot Study used in a graphical perception experiment by Javed et al. (2012) where findings from thepilots were used to find suitable levels for the factors included in the experiment.