https://max.pm/posts Mostly facts and logic. And, sometimes, unbridled emotion. en https://max.pm/posts/go-proxy-mess/ https://max.pm/posts/go-proxy-mess/ https://max.pm/posts/otree-ws-vuln/ https://max.pm/posts/otree-ws-vuln/ https://max.pm/posts/sequencing/ https://max.pm/posts/sequencing/ https://max.pm/posts/impuls-paternalism/ https://max.pm/posts/impuls-paternalism/ https://max.pm/posts/loans/ https://max.pm/posts/loans/ rB_0$ to amortize the loan over time, so that the loan is actually paid off at $N$.) The implication is that the unique possible payment schedule indeed starts off with a "high" interest portion that subsequently trails off. However, $\Pi_{n+1} - \Pi_n = r(P - rB_0)(1+r)^{n-1}$ is positive! Thus, it is correct that, over time, the portion paid to the principal increases. So the next time someone complains about banks’ “practice” of having you “pay interest first:” There is no practice. There is no policy. There is no choice. The payment schedule is not a decision anyone made. It is a mathematical necessity, as unavoidable as $2 + 2 = 4$. What if you tried to “pay principal first”? Suppose you demanded that your bank let you pay principal first, meaning you wanted $\Pi_n$ to start high and decrease over time, rather than the other way around. What would happen? For $\Pi_n = (P - rB_0)(1+r)^{n-1}$ to be decreasing, we would need $P - rB_0 \lt 0$, i.e., $P \lt rB_0$. But look at what this implies for the balance. After the first period, $B_1 - B_0 = rB_0 - P > 0.$ The balance is increasing. Your payment does not even cover the interest! This is called negative amortization, where the debt grows rather than shrinks. Far from paying the loan off faster, you’re falling further behind! Worse still, with $P \le rB_0$, the loan cannot be paid off in finite time. The boundary condition $B_N = 0$ becomes impossible to satisfy for any finite $N$. You would have to either increase your payment (back above $rB_0$, restoring the "interest-first" structure), extend the term to infinity (possible only for $P = rB_0$), or default. No bueno. And once again, this is simply just mathematics. No politician can save you from plain mathematics. In other words, “pay principal first” is not an alternative payment schedule. The assumptions that define a standard amortizing loan (a fixed rate, fixed term, fixed payment, balance paid off at maturity) mathematically require the interest-heavy-first structure. You cannot violate it without violating one of those assumptions. It is true that borrowers are often allowed to pay extra principal early (prepayments/curtailments). That changes the balance path and reduces total interest, which may of course be sensible. But it is not mathematically possible to merely increase the percentage of $P$ being spent on $\Pi_n$ without violating important assumptions of the model. Similarly, borrowers can often opt for interest-only payments to temporarily reduce payments. Once again, there is no free lunch: such arrangements require either higher payments later, an extended term, or a balloon payment at maturity. Other loan structures can violate other assumptions of the model, which may be individually acceptable, but the fundamental equations dictated by mathematics do not and cannot change. ]]> https://max.pm/posts/bright-leds/ https://max.pm/posts/bright-leds/ https://max.pm/posts/experimental-hazards/ https://max.pm/posts/experimental-hazards/ 30$, or similar, for the CLT to apply. (This is an oversimplification. It depends heavily on the underlying distribution’s skewness.) However that may be, this does not mean that the t-test would fail. There are two cases that should be distinguished here: Type I errors and Type II errors. For small $n$ and roughly equal group sizes, both the degrees of freedom, $\nu$, and the estimate of the standard deviation, $\sigma$, lead to such an unbelievably conservative distribution of t that I am confident to say that your test will not be oversized. REWARD! If you are the first one to send me any R function at the indicated position subject to the conditions in the code, with the script running without error, the test at the end being rejected at $\alpha = 10^{-12}$, then I will admit defeat, update this post accordingly, and send a charity of your choice US$100: Show R code Hide R code This offer was posted on 2026-01-02 and is still valid. Good luck. Second, with respect to Type II errors: if you ever have $n \lt 30$, that’s a You problem. Increase your sample size or improve your design more generally. That’s part of why power analyses are useful. Nonparametric tests’ null hypotheses are poorly understood A common claim about Wilcoxon-type tests is that they are tests on the median. In general, that is false. See also this. And this. If you are happy with making the additional assumptions required to reframe Wilcoxon-type tests as tests on the median, then why not simply make the minimal assumptions for the t-test (see above) and actually test something real? Generally speaking, Wilcoxon-type tests are tests on stochastic dominance. While in some very specific instances, stochastic dominance may be what matters (see below), in general it is not. By the way: even if nonparametric tests tested for the median—or if you’re willing to make the necessary assumptions—that would only indicate the difference in medians between your conditions, not the movement of a “baseline median” person to a different outcome given treatment. The latter interpretation requires additional assumptions (such as rank invariance, constant additive treatment effects, or other restrictions on the joint distribution of potential outcomes). More broadly, your theory would have to justify why medians matter at all. We know what t-tests do and that they have good properties t-tests are tests on the mean. Means (or, rather, ATEs) matter a whole lot over a broad bandwidth of economic theory. And that’s that. That t-tests have good properties was previously accomplished (warning: loud). Linear regression with HC3 standard errors (“OLS-HC3”) is even better than t-tests OLS is essentially an extension of the t-test. The t-test is just OLS with a binary treatment indicator. OLS generalizes this to multiple treatments, continuous controls, and factorial designs. It should be the natural framework for experimental analysis. But what about heteroskedasticity? That problem is basically solved. Just use HC3 standard errors. They have excellent finite-sample properties. And yes, they are better than HC0, HC1, or HC2. Just use HC3. In R: lmtest::coeftest(model, vcov = sandwich::vcovHC(model, type = "HC3")). In Stata: reg y t, vce(hc3). Note that Stata by default uses HC1 if you just specify robust, so don’t do that. OLS has many incredible benefits beyond robustness. You can easily include additional control variables to improve precision (see the ANCOVA section below). You can cluster standard errors by session, group, or any other unit. You can estimate multiple treatment effects simultaneously by using saturated regressions. Coefficients are directly interpretable. There are no convergence issues. Results are transparent and reproducible. OLS-HC3 should be the default workhorse for experimental economics. It just works. Note: there is a critical chapter on “robust” standard errors, including HC3, in Mostly Harmless Econometrics, Section 8.1. Their simulation results rely on very small sample sizes, unequal between treatments (literally $N_1 = 0.1 \cdot 30 = 3$ in one of the treatments) and should be viewed as extreme and purely pedagogical. Also, interestingly, the newer HC4 standard errors (but not HC5!), mitigate the issue. I attach R code below to replicate the results in their final column for the cases of (i) no and (ii) lots of heterogeneity. If you change FACTOR or TREATED, you can see how well HC3 in fact performs under even slightly more realistic scenarios despite low sample sizes. Show R code Hide R code Even the analysis of binary outcomes using OLS-HC3 is probably fine (with simple designs) A common concern is that when your outcome variable is binary (0 or 1), you “should” use logit or probit instead of OLS. This concern is largely misplaced for experimental work with simple designs. The linear probability model (LPM, just OLS with a binary dependent variable) has many advantages: (i) coefficients are directly interpretable as percentage point changes in probability; (ii) there are no convergence issues; (iii) it does not impose functional form assumptions about how treatment effects vary across the probability distribution; (iv) with HC3 standard errors, it is robust to heteroskedasticity (which is inherent with binary outcomes). The main remaining criticism of the LPM is that fitted values can fall outside $[0, 1]$. I have never seen fitted values matter in experimental economics. If you extrapolate, you must choose an appropriate data-generating process. For simple treatment comparisons, the LPM with HC3 standard errors is an excellent default choice. It is transparent, robust, and interpretable. It just works. More precisely, the LPM is valid when all right-hand-side variables are categorical and fully interacted. Once continuous covariates enter the model, functional form issues can arise. But for testing differences between treatments, where the key regressor is just a dummy variable, this is not a concern. Logit and probit models should be reserved for cases where (i) you have strong theoretical reasons to impose a specific functional form (such as from utility theory), or (ii) you are working with observational data where extrapolation matters. One final note: if you do use logit or probit, report marginal effects, not raw coefficients. Raw coefficients from nonlinear models are nearly impossible to interpret and compare across studies. Marginal effects at the mean (or average marginal effects) restore interpretability. Interactions (and why you probably shouldn’t care about them) As I mention below in the section on factorial designs, interactions are fundamentally model-dependent. An interaction that is significant in OLS may be insignificant in logit, or vice versa. Moreover, interactions are almost always underpowered ex ante. In factorial designs, as discussed above, you should include the interaction term in your regression to avoid functional form misspecification, but you should typically focus on main effects or simple effects for interpretation. The interaction coefficient itself is rarely of interest. Feel free to just eliminate conditions where multiple treatments are active if you don’t need the resulting comparisons. Small design changes can vastly improve statistical inference One of the highest-return design modifications you can make is to collect a baseline measure of your outcome variable before randomizing participants into treatment. Then, include this baseline measure as a control variable in your analysis. This approach is called ANCOVA (analysis of covariance) and it can dramatically improve statistical power. Consider the standard regression for a randomized experiment: \begin{equation} y_i = \beta_0 + \tau T_i + \gamma \pmb{X} + \varepsilon_i \end{equation} Here, $y_i$ is your outcome, $T_i$ is the treatment indicator, and $\pmb{X}$ represents any additional control variables (demographics, etc.). Now suppose you collect a baseline measure $y_i^0$ of the outcome before treatment assignment. You can then estimate: \begin{equation} y_i = \beta_0 + \beta_1 y_i^0 + \tau T_i + \gamma \pmb{X} + \varepsilon_i \end{equation} This helps because $y_i^0$ absorbs individual-level variation in the outcome. If people differ substantially in their baseline levels, the inclusion of $y_i^0$ reduces the residual variance $\text{Var}(\varepsilon_i)$. This directly shrinks the standard error of $\hat{\tau}$, increasing your statistical power without requiring a larger sample! The gains can be enormous. But: timing matters. The baseline measure must be collected before randomization. However, any pre-treatment variable that is correlated with the outcome will help. For example, if your outcome is post-treatment donations, a baseline measure of past donations or general prosociality will still improve precision. Measurement error in $y_i^0$ attenuates $\beta_1$ but does not bias $\hat{\tau}$, because $T_i$ is randomized and thus by construction uncorrelated with $\varepsilon_i$. Summing up, if your experiment allows for it, always collect a baseline measure. There is essentially no downside. Preregister the inclusion of $y_i^0$ and always include it in your main specification. Carefully design comparisons in factorial designs (e.g., 2×2) Show R code Hide R code Suppose you have two factors, $g$ and $h$. Both are dummy variables (1 if “turned on” and 0 otherwise). Participants get iid randomly assigned to $g$ and/or $h$. As is clear, this is a 2×2 between-subjects design. Moreover, you have your outcomes, y. What can you do? There are three standard effects in 2×2 designs: main effects, simple effects, and interaction effects. Economists are inherently (and rightly) suspicious of interaction effects, and thus I will not be covering them. A core challenge with interpreting interactions is that they work only for certain outcome variables and models. With binary outcome variables, for example, an interaction that is not significant in the linear probability model/with OLS may be significant in a logit model, a probit model, both, or neither. It is a huge unresolved and likely unresolvable mess. Never rely on the interaction in a factorial design unless you fully understand the model to be used on the outcome. Simple effects are simple indeed; they refer to the effect of one factor at a fixed level of the other. In R, just do lm(y ~ h, data = df[df$g == 1, ]) (with optional control variables, see above) and get lunch. Your day is completed. Main effects are not so simple. The main effect of, say, $g$, requires you to average over $h$ with some weights. If these weights can be derived from a policy question or your theoretical framework, great! Otherwise, just use equal weights. Denote by $\mu_{g,h}$ the population mean in some factorial treatment $g, h$. Then, the main effects of $g$ and $h$ are defined as follows: \begin{align} \tau_g^{\text{M}} = \left[ w_1 \mu_{1,0} + (1-w_1) \mu_{1,1} \right] - \left[ w_1 \mu_{0,0} + (1-w_1) \mu_{0,1} \right]\\ \tau_h^{\text{M}} = \left[ w_2 \mu_{0,1} + (1-w_2) \mu_{1,1} \right] - \left[ w_2 \mu_{0,0} + (1-w_2) \mu_{1,0} \right] \end{align} In the following, I assume $w_1 = w_2 = \frac{1}{2}$. As I argued above, linear regressions are an excellent method to analyze your experiment. How, then, can we get $\tau_g^{\text{M}}, \tau_h^{\text{M}}$ from a neat linear regression? Unfortunately, this linear model is simply wrong: \begin{equation} y_i = \beta_0 + \beta_1 g_i + \beta_2 h_i + \beta_3 g_i h_i + \varepsilon_i \end{equation} Crucially, with main effects, it is not correct to just throw your (binary) dummy treatment indicators into the linear model as-is. That would only be correct if you did not use an interaction term in your linear model. Given my skepticism about interaction terms, you may ask why I would ever propose using one of these! The reason is simple: including the interaction term makes your model saturated. It can exactly recover all four cell means without imposing any functional form restrictions. If you omit the interaction term, you assume additivity; if this assumption is wrong, your main effect estimates will be biased. Including the interaction term protects against this misspecification. Therefore, you should always include the interaction term but (probably) ignore the coefficient on it. In general, $\beta_1 \neq \tau_g^{\text{M}}$ and $\beta_2 \neq \tau_h^{\text{M}}$. Why is that? Simple: \begin{equation} \frac{\partial y_i}{\partial g_i} = \beta_1 + \beta_3 h_i \label{marginal} \end{equation} This marginal effect depends on the value of $h_i$! Only at $h_i = \frac{1}{2}$ does the marginal effect equal the main effect. So, we must transform $g_i, h_i$ as follows: \begin{align} g'_i = g_i - \frac{1}{2}\\ h'_i = h_i - \frac{1}{2} \end{align} This effect coding or deviation coding ensures that the marginal effect is equal to the main effect, since we can now ignore the final term in the equivalent of Equation \eqref{marginal}. The following linear model recovers the main effects: \begin{equation} y_i = \beta_0 + \tau_g^{\text{M}} g'_i + \tau_h^{\text{M}} h'_i + \Xi g'_i h'_i + \varepsilon_i \end{equation} Needless to say, you can include further control variables as usual (and should ignore $\Xi$). However, transforming dummies is crucial—as long as you do that, your coefficients are meaningful! See here and here for excellent references. Show R code for 2×2×2 designs Hide R code for 2×2×2 designs Null effects can in fact be quantified A common reaction to a non-significant result is utter despair. That reaction is easily wrong or at least premature. A p-value above 0.05 tells you that you failed to reject the null hypothesis. It does not tell you that the effect is zero, small, or negligible. It says nothing about effect size. A critical distinction must be made between underpowered null results (which are in general not so valuable) and well-powered, tightly bounded null results (which are valuable). If your study had 20% power to detect a small effect and you find $p = 0.5$, you have learned essentially nothing. Your confidence interval will be wide, spanning large positive and negative effects. Contrarily, if your study has 90% power and you find $p = 0.5$ with a confidence interval of $[-0.05, 0.15]$ in standardized units, you have learned something important: the effect, if it exists at all, is small. Let’s not mix both of these cases! Null results can be valuable, but only if they are informative. An informative null result rules out effects larger than some threshold. To claim an informative null in a frequentist framework, you must demonstrate that your confidence interval is sufficiently narrow. There are two main approaches to quantifying null effects: equivalence testing (TOST) and Bayesian methods. Both allow you to make positive claims about the absence or negligibility of an effect, rather than merely failing to reject the null. Using TOST Show R code Hide R code The two one-sided tests (TOST) procedure tests whether your effect lies within a pre-specified equivalence region. Instead of testing $H_0: \tau = 0$, you test whether $\tau$ is practically equivalent to zero by checking if the confidence interval for $\tau$ lies entirely within some interval $[-\Delta, \Delta]$, where $\Delta$ is the smallest effect size you consider meaningful. The conventional choice is $\Delta = 0.2$ in standardized units (Cohen’s $d$), though you should justify this threshold based on your research context. A narrower equivalence region (say, $\Delta = 0.1$) makes a stronger claim but requires more statistical power. The TOST procedure uses a 90% confidence interval, which corresponds to two one-sided tests at $\alpha = 0.05$. Here is an example (from the code above): To assess whether the effect is practically negligible, we report the 90% confidence interval, which corresponds to a two one-sided tests (TOST) equivalence procedure at $\alpha = 0.05$ (Lakens et al., 2018). The estimated coefficient is $-0.032$ ($\text{SE} = 0.072$), yielding a 90% CI of $[-0.150, 0.086]$. Given the pooled outcome standard deviation of $1.329$, this corresponds to $[-0.113, 0.065]$ in standardized units (Cohen’s $d$). Using the conventional threshold of $|d| = 0.2$ as the smallest effect size of interest, the 90% CI lies entirely within the equivalence region $[−0.2, 0.2]$, supporting the conclusion of practical equivalence. Using Bayesian t-tests Show R code Hide R code NEW: Open web app My personal intuitionist view of evidence is as follows: any study should yield one of three conclusions: evidence for X, evidence for Y, or indeterminate. Bayesian tests deliver on all three possibilities. A Bayes Factor (BF) can provide evidence for the null, evidence for the alternative, or remain inconclusive ($\text{BF} \approx 1$). Frequentist methods, by contrast, do not distinguish between evidence for $H_0$ and poor data. This is why I prefer Bayesian approaches for quantifying null effects, though TOST serves a similar purpose when properly powered. However, Bayesian methods come with batteries included. As I like to say: “Bayesian methods solve all problems, for free.” It’s true. Really! Simply put, Bayesian methods provide an alternative approach by computing a Bayes Factor, which quantifies the relative evidence for the null hypothesis versus the alternative. A $\text{BF}_{01} > 3$ suggests moderate evidence for the null, while $\text{BF}_{01} > 10$ suggests strong evidence. Conversely, $\text{BF}_{01} \lt 1/3 \equiv 1 / \text{BF}_{10}$ suggests at least moderate evidence for an effect. Read more here. Bayesian t-tests (e.g., as implemented in the BayesFactor package in R) specify a prior distribution on effect sizes and compute how much the data update your beliefs. The advantage of this approach is that you can actually claim evidence for the null, not just failure to reject it. The disadvantage is that results depend on your choice of prior. Therefore, always use the now-standard default prior of $\text{Cauchy}\left(0, \frac{\sqrt{2}}{2}\right)$ and conduct robustness analyses. Here is an example (from the code above, same data as with the TOST example): We conducted a Bayesian independent samples t-test (Rouder et al., 2009) using a default prior with scale parameter $r = \frac{\sqrt{2}}{2} \approx 0.707$ (Cauchy distribution centered at zero). The Bayes Factor strongly favored the null hypothesis, with $\text{BF}_{01} = 15.0$ (equivalently, $\text{BF}_{10} = 0.067$), indicating that the observed data are 15 times more likely under the null hypothesis of no effect than under the alternative hypothesis. This constitutes strong evidence for the absence of an effect. To assess robustness to prior specification, we conducted sensitivity analyses with alternative scale parameters: $r = 0.5$ yielded $\text{BF}_{10} = 0.094$ ($\text{BF}_{01} = 10.7$); $r = 1.0$ yielded $\text{BF}_{10} = 0.047$ ($\text{BF}_{01} = 21.1$); and $r = 2.0$ yielded $\text{BF}_{10} = 0.024$ ($\text{BF}_{01} = 42.2$). Across all specifications, we obtained consistent strong evidence favoring the null hypothesis ($\text{BF}_{01} > 10$ in all cases), demonstrating that our conclusion is robust to reasonable prior choices. In practice, TOST and Bayesian methods often agree qualitatively. If you have a well-powered study with tight confidence intervals, TOST will show equivalence and the Bayes Factor will favor the null (as in the example above). If your study is underpowered, neither method will save you: TOST will fail to show equivalence and the Bayes Factor will be inconclusive (close to 1). The lesson is simple: power matters for null results just as much as for positive results. Design your study properly. Know what you are doing, measuring and estimating Any experiment starts with a theory of human behavior. This theory may not be mathematical, but it is always a statement about counterfactual behavior. The section title promises three requirements: know what you are doing (treatment design), know what you are measuring (outcome elicitation), and know what you are estimating (estimands). All three must align with your theory’s objects. Know what you are doing For example, suppose that you want to study workers’ effort when provided with information about coworkers’ effort. Now this is clearly an experiment about workers’ beliefs. These beliefs are changed through information, and then those beliefs are (probably) thought to induce a response in effort. I have talked more above about some recommendations for such experiments, but for now let us just focus on treatment design. Putting aside questions of deception, a simple experiment could tell some participants (i) that their group of coworkers had an average effort level of $e_1$ and some other participants (ii) that their group of coworkers had an average effort level of $e_2$, where $e_2 \lt e_1$. Do you see a problem with this design? Show problem Hide problem This design is built on the assumption that workers’ effort level is shaped by beliefs about the average effort of others. But it could well be that their effort responds to the minimum effort, the maximum effort, or any other statistic of the distribution. If your theory proves that only beliefs about averages should matter, then such a design is wholly appropriate. And even if people’s effort is in fact a function of the minimum, the minimum could still be correlated with the average, so the design may “work” accordingly. Still, beliefs are extraordinarily rich objects, and any particular experimental configuration of beliefs must grapple with that complexity. As Einstein put it: “theory decides what we can observe.” So true! Formal mathematical theory has the great feature that it forces you to make explicit assumptions. Formal theory constrains an experimenter’s degrees of freedom. This is why theory is invaluable even when not strictly necessary: it disciplines experimental design. Know what you are measuring Your outcome measures must be as close as possible to theoretically relevant objects. There are important exceptions: sometimes the theoretically relevant object (such as a preference parameter) cannot be directly elicited as an outcome measure. Simple tasks like binary choices, coupled with a structural model, can nonetheless be highly informative about it. Theory still dictates what is relevant. It just also dictates the mapping from observable outcomes to theoretical objects. In the absence of formal theory, or if theory makes no predictions for a particular object, your planned analyses can serve as a reduced-form guide to what you need to measure. A key principle: comparisons are only valid when the objects being compared are of the same kind. The null hypothesis must be a sensible benchmark. Suppose you want to compare first-order beliefs (what I believe about others) with second-order beliefs (what I believe others believe about others). You elicit the first-order belief by asking: “What is the average effort of others?” Now, how should you elicit the second-order belief? An invalid approach would be to ask: “What is the most common belief about the average effort of others?” This compares a mean (first-order) with a mode (second-order). These are two different statistics that need not coincide even under rationality. A valid approach could ask: “What is the average belief about the average effort of others?” Now both elicitations concern means. Under common priors and rational expectations, the first-order and second-order beliefs should coincide. Any divergence could reveal false consensus, projection bias, or asymmetric updating. Crucially, the comparison is valid in principle because the null hypothesis (equality under rationality) is a meaningful benchmark. Know what you are estimating We have discussed estimands in greater detail above. For now, keep the following principle in mind, which I call the Fundamental Law of Experimental Economics: If theory predicts a monotonic individual response, a zero average treatment effect (ATE) implies zero individual effects. This law holds because the mean cannot hide offsetting responses when all individual effects have the same sign. Under random assignment, the ATE therefore directly tests the theory. This is a precise statement: the ATE is an expectation (an integral over the distribution of individual types) that identifies the causal effect of a stimulus. Random assignment “marginalizes out” unobservable theoretical parameters, leaving only the predicted effect. The Fundamental Law immediately suggests how to design good experiments: isolate behavioral factors that, under theory, predict a monotonic change. Such factors deliver clean comparative statics. Contrarily, if theory permits heterogeneous signs (some individuals respond positively, others negatively) a zero ATE is uninformative. It could reflect no effects, or it could reflect large offsetting effects. Without further restrictions, you cannot distinguish these cases. The upshot is that experimental design and statistical analysis must be in concordance. The estimands of interest must directly test the predicted monotonic relationships that your theory delivers. This is what it means to know what you are estimating. Radical openness is likely costless or a free lunch (if you start early) Transparency is free if you do things properly from the beginning. Do good theory. Write clean analysis code. Organize your files sensibly. When the time comes to publish a replication package, you will have virtually no extra work. The cost is zero. Actually, the cost is negative, because good practices save you time during the project itself! Better yet: just upload sanitized data immediately after your experimental sessions to GitHub (if permitted). Make your analysis scripts public from day one. There is little reason to wait. If your IRB or data agreements prohibit immediate sharing, fine. Otherwise, default to radical openness. And no, you will not be “scooped.” Your project is, in all likelihood, not that interesting. Preanalysis plans and preregistrations are also valuable. They protect you from accusations of p-hacking. They force you to think clearly about your design and analysis before seeing the data. They improve the credibility of your findings and they make our whole science better. Also, by committing to analyses, sample sizes, and the like beforehand you can remove the need for later decisions. Just do it. ]]> https://max.pm/posts/bayes-factors/ https://max.pm/posts/bayes-factors/ 100$ $> 0.99$ Extremely strong evidenceB A As per Jeffreys (tradition has it). B In favor of the hypothesis under consideration. Relationship to likelihood ratio tests The classical likelihood ratio test statistic is \begin{equation} \Lambda = \frac{\sup_{\theta \in \Theta_1} P(D \,\vert\, \theta)}{\sup_{\theta \in \Theta_0} P(D \,\vert\, \theta)} \end{equation} where $\Theta_0, \Theta_1$ are the parameter spaces under the null and alternative. This ratio compares the best-fitting parameter values. The Bayes Factor instead compares average likelihoods: \begin{equation} BF_{10} = \frac{\int P(D \,\vert\, \theta) \, p(\theta \,\vert\, H_1) \, d\theta}{\int P(D \,\vert\, \theta) \, p(\theta \,\vert\, H_0) \, d\theta} \end{equation} The averaging penalizes models with diffuse priors over large parameter spaces: probability mass wasted on poor-fitting parameter values drags down the marginal likelihood. This provides an automatic Occam's razor absent from $\Lambda$. Decomposing Bayes Factors In some cases, there is no single data set $D$ that can be used to evaluate the hypotheses, but rather multiple “evidence” $E_1 , E_2, \dots, E_m$ such that $E \equiv \bigcap_{i=1}^{m} E_{i}$ is the totality of evidence. Rewriting Equation \eqref{bayes} in terms of evidence, we get \begin{equation} P(H \,\vert\, E) = \frac{P(H) P(E \,\vert\, H)}{P(E)}. \end{equation} Using the chain rule, for $m=2$ it holds that \begin{equation} P(E \,\vert\, z) = P(E_1 \,\vert\, z) P(E_2 \,\vert\, E_1, z) \end{equation} for any conditioning variable, $z$. More generally, \begin{equation} P(E \,\vert\, z) = P(E_1 \,\vert\, z) \prod_{i=2}^{m} P(E_i \,\vert\, E_1, \dots, E_{i-1}, z). \end{equation} The Bayes Factor can thus be decomposed as \begin{align*} \frac{P(E \,\vert\, H_1)}{P(E \,\vert\, H_2)} &= \frac{P(E_1 \,\vert\, H_1) \prod_{i=2}^{m} P(E_i \,\vert\, E_1, \dots, E_{i-1}, H_1)}{P(E_1 \,\vert\, H_2) \prod_{i=2}^{m} P(E_i \,\vert\, E_1, \dots, E_{i-1}, H_2)}\\ &= \frac{P(E_1 \,\vert\, H_1)}{P(E_1 \,\vert\, H_2)} \prod_{i=2}^{m} \frac{P(E_i \,\vert\, E_1, \dots, E_{i-1}, H_1)}{P(E_i \,\vert\, E_1, \dots, E_{i-1}, H_2)}\\ &= BF_{12}^{E_1} \cdot BF_{12}^{E_2 \, \vert \, E_1} \cdot \dots \cdot BF_{12}^{E_m \, \vert \, E_1, \dots, E_{m-1}} \end{align*} This decomposition is useful when evidence arrives sequentially or when different pieces of evidence have qualitatively different sources. Instead of computing a single likelihood ratio over all evidence at once, you can update beliefs incrementally. Each factor $BF_{12}^{E_i \, \vert \, E_1, \dots, E_{i-1}}$ measures how much the $i$-th piece of evidence favors $H_1$ over $H_2$, given what was already known. In practice, this matters when some evidence is easier to evaluate than others, or when you want to diagnose which pieces of evidence drive the overall conclusion. A large overall Bayes factor might be dominated by a single $E_i$, or it might accumulate from many modest contributions. The decomposition makes this transparent. It also helps when combining evidence from heterogeneous sources (e.g., experimental data and observational data) where assuming independence would be wrong, but the conditional structure is tractable. When the pieces of evidence are conditionally independent given the hypothesis (when $P(E_i \mid E_1, \dots, E_{i-1}, H) = P(E_i \mid H)$ for all $i$) the decomposition simplifies to a product of unconditional Bayes factors: \begin{equation} \frac{P(E \,\vert\, H_1)}{P(E \,\vert\, H_2)} = \prod_{i=1}^{m} \frac{P(E_i \,\vert\, H_1)}{P(E_i \,\vert\, H_2)} = \prod_{i=1}^{m} BF_{12}^{E_i} \end{equation} This is convenient but rarely justified in practice. Evidence from the same domain or measurement process is typically correlated, and treating dependent evidence as independent can inflate confidence. ]]> https://max.pm/posts/r_not_yours/ https://max.pm/posts/r_not_yours/ https://max.pm/posts/framework_13_debian/ https://max.pm/posts/framework_13_debian/