Many questions formulated by legal scholars address a causal relationship between the law and human behavior: What is the effect of “teaser rates” on credit card debt? Will delegation of decision-making diminish the “stickiness of default rules”? Does a key information document improve retail investors’ understanding of investment products? Behavioral law and economics can help answering these questions, as it provides predictions about people’s behavior in a legally relevant context. Its increasing use in policy-making and law calls for an understanding of how these predictions are formulated and tested. Laboratory experiments are an important empirical method employed to this end. Thus, it comes at no surprise that legal scholars are increasingly interested, and even involved, in experimental research. In this blog post, we take a look at the advantages of this method (1), its possible uses in legal scholarship (2), and the most pressing concern over its use – the external validity of experimental results (3).
1. The experimental advantage
The main advantage of the experimental method is that it allows testing causal relationships between rules and human reactions. This is possible due to the characteristics of laboratory experiments. First, in a laboratory experiment a researcher implements situations (experimental conditions) that differ with respect to only one feature. For instance, some participants can be given a possibility to delegate their decision to a third-person while others make their decisions themselves. Second, participants are assigned to these conditions randomly.
These two steps make sure that any differences observed in the behavior of these two groups can be attributed to this one feature that differed between experimental conditions and not to any other situational changes or particular characteristics of individuals participating in one or the other condition. Importantly, the results of an experimental study are analyzed by testing hypotheses formulated before a study is conducted. Simply put, this way we know the probability that the observed differences occurred because of chance or because people indeed behave differently in these two situations.
2. The uses of experiments in legal scholarship
Legal scholarship can exploit the experimental advantage in at least three ways:
a) To identify human behavior in need of a policy intervention
To begin with, experiments can help us understand whether there actually is a problem that might require a policy intervention. An example of such an insight is the research on hyperbolic discounting showing that humans tend to value present gains or losses stronger than future gains or losses. This leads, for instance, to underestimating the cost of future debt, a bias that might be exploited by credit card companies through “teaser rates”. The legislator has reacted to this – in Europe with the Directive 2008/48/EC on credit agreements for consumers, and in the US with the CARD Act. Notwithstanding that it is contested whether these reactions suffice (see Bar-Gill and Rubb 2011), we have a situation, where experimental research has contributed tremendously to our understanding of a problem. But, this is only the first step. Whether the descriptive account of a problem justifies legal interventions, and whether the policy interventions, e.g., nudges, can be reconciled with the law are then questions that legal scholars have to deal with (see also the debate on Verfassungsblog).
b) To test whether the existing institutions could be implemented to solve a problem
The behavior that has been identified by behavioral law and economics as in need of a legal intervention, could already be sufficiently mitigated by existing institutions. Laboratory experiments are used to test whether this could indeed be the case. For instance, the endowment effect is yet another behavioral phenomenon that has been found troubling by legal scholars. It refers to situations when “people […] demand much more to give up an object than they [are] willing to pay to acquire it”. It has been subject to multiple experimental studies and is one of the common explanations for the “stickiness of default rules”, i.e., people’s unwillingness to contractually change the default provision stipulated in law (Korobkin 1998). Legal interventions have been suggested to help people overcome such a bias, e.g., by forcing people to choose their privacy settings when browsing the internet (on “choice-requiring paternalism”, Sunstein 2014). Interestingly, Jennifer Arlen and Stephan Tontrup showed that existing institutions can help counteract the endowment effect. They tested the behavioral propositions experimentally showing that debiasing institutions are found in collective decision mechanisms like voting, and the delegation of decisions to agents. So, while there may be a problem that can trigger a need for an intervention, the instruments to tackle it may already exist. Think of a company statute that require some form of collective decision-making. Or, somewhat simplified: Think of the privacy settings that you might have entrusted to an IT department. Here, the IT department has worked for you as an agent, and you are happy to leave the choice to them. Accordingly, one can argue for regulatory restraint, where such debiasing options exist.
c) To design and test the effectiveness of new legal rules
Laboratory experiments can also help test the effectiveness of new legal rules. An important area is consumer protection in investment decisions. The report by Nick Chater, Steffen Huck and Roman Inderst reviews existing research on consumers biases and heuristics that might affect their investment choices. It further reports the results of experiments testing the effectiveness of various interventions designed to overcome these biases and improve consumer investment decisions. These studies contributed to the development of Key Information Documents (KID) for Packaged Retail and Insurance-based Investment Products (PRIIPs). Importantly, the underlying EU Regulation explicitly refers to the need to take “ongoing research into consumer behaviour” into account when designing the ways of presenting information to consumers. And indeed, further experiments have been conducted showing how presentation of information matters for investment decisions (see Hillenbrand and Schmelzer 2017).
Interestingly, even the assessment of experienced traders will depend on the
way information on shares is presented (see Glaser, Iliewa, Weber 2016). All these studies are great examples of how laboratory experiments can be employed to test and design legal interventions.
3. Grappling with external validity
Yet, the much appreciated controlled environment of laboratory experiments can be a double-edged sword. Specifically, laboratory experiments have been criticized for a lack of external validity, i.e., a difficulty in transferring the results observed in the laboratory to the real world, as the latter is much more complex and contextually-rich. A related critique points to the predominant use of student samples in laboratory experiments. There is a fear that we cannot extrapolate these finding to a broader population.
The sampling question has to be put into perspective: Student samples are not a problem when a general hypothesis is tested, i.e., a hypothesis that should apply to the behavior of people in general (more on this by Falk and Heckmann 2009). However, student samples can indeed be a potential pitfall, if the experiment claims to say something about a certain trait of or the behavior of, e.g., judges. Thus, whether extrapolating experimental findings to a broader population is justified will depend on the match between the research question that the study addresses and the design of the experiment. A further argument for external validity is that the laboratory setting is artificial but not fake: Subjects earn real money that is paid out exactly as the experimenter has communicated to them and depending on their own decision-making.
But, indeed, conclusions drawn by legal scholars should not be based on experiments alone, much less on a single one. Legal scholars relying on experimental results should avoid cherry picking studies that support a particular legal argument or intervention, when there are in fact studies that point to a different direction. Ideally, the results from lab experiments should also be contrasted with results generated using other empirical methods (for an overview, see van den Boom, Desmet, Mascini 2018). Further, experimental effects should be replicated to be considered reliable, and the need for replication has been recognized in the discipline (Hoeppner and Hippel 2018; Irvine, Hoffmann, Wilkinson-Ryan 2018). Dealing with all of these concerns requires a basic understanding of the experimental method. Thus equipped, experiments become one more powerful instrument in the toolbox of empirical legal scholarship.
This blog post is based on the paper Experimentelle Ökonomik im Recht from the peer-reviewed German law journal Rechtswissenschaft.[:en]Many questions formulated by legal scholars address a causal relationship between the law and human behavior: What is the effect of “teaser rates” on credit card debt? Will delegation of decision-making diminish the “stickiness of default rules”? Does a key information document improve retail investors’ understanding of investment products? Behavioral law and economics can help answering these questions, as it provides predictions about people’s behavior in a legally relevant context. Its increasing use in policy-making and law calls for an understanding of how these predictions are formulated and tested. Laboratory experiments are an important empirical method employed to this end. Thus, it comes at no surprise that legal scholars are increasingly interested, and even involved, in experimental research. In this blog post, we take a look at the advantages of this method (1), its possible uses in legal scholarship (2), and the most pressing concern over its use – the external validity of experimental results (3).
1. The experimental advantage
The main advantage of the experimental method is that it allows testing causal relationships between rules and human reactions. This is possible due to the characteristics of laboratory experiments. First, in a laboratory experiment a researcher implements situations (experimental conditions) that differ with respect to only one feature. For instance, some participants can be given a possibility to delegate their decision to a third-person while others make their decisions themselves. Second, participants are assigned to these conditions randomly.
These two steps make sure that any differences observed in the behavior of these two groups can be attributed to this one feature that differed between experimental conditions and not to any other situational changes or particular characteristics of individuals participating in one or the other condition. Importantly, the results of an experimental study are analyzed by testing hypotheses formulated before a study is conducted. Simply put, this way we know the probability that the observed differences occurred because of chance or because people indeed behave differently in these two situations.
2. The uses of experiments in legal scholarship
Legal scholarship can exploit the experimental advantage in at least three ways:
a) To identify human behavior in need of a policy intervention
To begin with, experiments can help us understand whether there actually is a problem that might require a policy intervention. An example of such an insight is the research on hyperbolic discounting showing that humans tend to value present gains or losses stronger than future gains or losses. This leads, for instance, to underestimating the cost of future debt, a bias that might be exploited by credit card companies through “teaser rates”. The legislator has reacted to this – in Europe with the Directive 2008/48/EC on credit agreements for consumers, and in the US with the CARD Act. Notwithstanding that it is contested whether these reactions suffice (see Bar-Gill and Rubb 2011), we have a situation, where experimental research has contributed tremendously to our understanding of a problem. But, this is only the first step. Whether the descriptive account of a problem justifies legal interventions, and whether the policy interventions, e.g., nudges, can be reconciled with the law are then questions that legal scholars have to deal with (see also the debate on Verfassungsblog).
b) To test whether the existing institutions could be implemented to solve a problem
The behavior that has been identified by behavioral law and economics as in need of a legal intervention, could already be sufficiently mitigated by existing institutions. Laboratory experiments are used to test whether this could indeed be the case. For instance, the endowment effect is yet another behavioral phenomenon that has been found troubling by legal scholars. It refers to situations when “people […] demand much more to give up an object than they [are] willing to pay to acquire it”. It has been subject to multiple experimental studies and is one of the common explanations for the “stickiness of default rules”, i.e., people’s unwillingness to contractually change the default provision stipulated in law (Korobkin 1998). Legal interventions have been suggested to help people overcome such a bias, e.g., by forcing people to choose their privacy settings when browsing the internet (on “choice-requiring paternalism”, Sunstein 2014). Interestingly, Jennifer Arlen and Stephan Tontrup showed that existing institutions can help counteract the endowment effect. They tested the behavioral propositions experimentally showing that debiasing institutions are found in collective decision mechanisms like voting, and the delegation of decisions to agents. So, while there may be a problem that can trigger a need for an intervention, the instruments to tackle it may already exist. Think of a company statute that require some form of collective decision-making. Or, somewhat simplified: Think of the privacy settings that you might have entrusted to an IT department. Here, the IT department has worked for you as an agent, and you are happy to leave the choice to them. Accordingly, one can argue for regulatory restraint, where such debiasing options exist.
c) To design and test the effectiveness of new legal rules
Laboratory experiments can also help test the effectiveness of new legal rules. An important area is consumer protection in investment decisions. The report by Nick Chater, Steffen Huck and Roman Inderst reviews existing research on consumers biases and heuristics that might affect their investment choices. It further reports the results of experiments testing the effectiveness of various interventions designed to overcome these biases and improve consumer investment decisions. These studies contributed to the development of Key Information Documents (KID) for Packaged Retail and Insurance-based Investment Products (PRIIPs). Importantly, the underlying EU Regulation explicitly refers to the need to take “ongoing research into consumer behaviour” into account when designing the ways of presenting information to consumers. And indeed, further experiments have been conducted showing how presentation of information matters for investment decisions (see Hillenbrand and Schmelzer 2017).
Interestingly, even the assessment of experienced traders will depend on the
way information on shares is presented (see Glaser, Iliewa, Weber 2016). All these studies are great examples of how laboratory experiments can be employed to test and design legal interventions.
3. Grappling with external validity
Yet, the much appreciated controlled environment of laboratory experiments can be a double-edged sword. Specifically, laboratory experiments have been criticized for a lack of external validity, i.e., a difficulty in transferring the results observed in the laboratory to the real world, as the latter is much more complex and contextually-rich. A related critique points to the predominant use of student samples in laboratory experiments. There is a fear that we cannot extrapolate these finding to a broader population.
The sampling question has to be put into perspective: Student samples are not a problem when a general hypothesis is tested, i.e., a hypothesis that should apply to the behavior of people in general (more on this by Falk and Heckmann 2009). However, student samples can indeed be a potential pitfall, if the experiment claims to say something about a certain trait of or the behavior of, e.g., judges. Thus, whether extrapolating experimental findings to a broader population is justified will depend on the match between the research question that the study addresses and the design of the experiment. A further argument for external validity is that the laboratory setting is artificial but not fake: Subjects earn real money that is paid out exactly as the experimenter has communicated to them and depending on their own decision-making.
But, indeed, conclusions drawn by legal scholars should not be based on experiments alone, much less on a single one. Legal scholars relying on experimental results should avoid cherry picking studies that support a particular legal argument or intervention, when there are in fact studies that point to a different direction. Ideally, the results from lab experiments should also be contrasted with results generated using other empirical methods (for an overview, see van den Boom, Desmet, Mascini 2018). Further, experimental effects should be replicated to be considered reliable, and the need for replication has been recognized in the discipline (Hoeppner and Hippel 2018; Irvine, Hoffmann, Wilkinson-Ryan 2018). Dealing with all of these concerns requires a basic understanding of the experimental method. Thus equipped, experiments become one more powerful instrument in the toolbox of empirical legal scholarship.
This blog post is based on the paper Experimentelle Ökonomik im Recht from the peer-reviewed German law journal Rechtswissenschaft.
Distributed under the Creative Commons License CC BY 4.0
