Luck Control

Problem
Chance is sometimes a major factor for certain tasks, such as visual search. For example, if the participant is searching for a target in a collection of distractors by clicking on each potential target to find out if it is the correct one, they may get lucky and immediately pick the correct target (or unlucky and always pick the correct target last). This renders the distractors largely ineffective.

Solution
The common solution is to leave issues such as this to random chance in the knowledge that they will even out over the course of the experiment, but sometimes luck may have too large an impact to be ignored. In such situations, the solution is to limit the impact of luck by explicitly controlling discovery order without the knowledge of the participants. For example, if there are five potential targets to pick and only one of them is the correct answer, add an experimental factor D with values 1 to 5 that says which of the five targets is the correct one. Whether a potential target is the right one or not is determined when the participant actually clicks on it (given earlier clicks). Each participant will thus be lucky (D = 1) and unlucky (D = 5) once per condition, and all other levels of chance in-between.

Consequences
Using Luck Control requires an additional factor to be added to the experimental design, which can sometimes be problematic for experiments that already have a large number of conditions. In addition, sometimes the number of possible outcomes is too large to model directly using a factor; in such situations, you may define intervals of outcomes as "easy", "medium", and "difficult" in terms of the impact of random chance (corresponding to, for example, discovery order 1-5, 6-10, and 11-15). Finally, Luck Control can only be used in situations where the determination of what potential target is the correct one can be performed on the spot, and not when setting up the trial.

A danger with this approach is that participants may suspect that luck is being controlled. However, if done correctly, this knowledge should not impact performance; the decision of which potential target is the correct one is lazily resolved.

Examples
A form of Luck Control is used in many experiments that include factors for the difficulty of a trial. However, to our knowledge, explicitly controlling discovery was first proposed by Pietriga et al. (2007) in their operationalization of multi-scale search (adopted by Javed et al. (2012)).