Validating experiments

Every day, people face decisions in domains as diverse as choosing among shampoos, stocks, medical treatments, and friends.  When people have not learned what to do through trial and error, they need a suite of generally applicable decision-making skills.  These include extracting relevant information, applying general values in specific settings, and integrating these pieces with a coherent decision rule.¹

Behavioral decision research has often found deficiencies in these skills.  These limits are reflected in such phenomena as preference reversals, anchoring and (insufficient) adjustment, sunk costs effects, poorly calibrated confidence judgments, and hindsight bias.¹  Investigators disagree about the level of external validity, pointing to the unnatural tasks and artificial environments used in the lab.

Accounts of decision-making processes typically point to four fundamental skills:  assessing beliefs; assessing values; combining beliefs and values in order to identify choices; and having a meta-cognitive understanding of one's abilities.¹

Expert Choice Inc area validation exercise

Saaty's validation exercises