4.4 Conclusion

This chapter was about what happens when the regressors misbehave, and the two ways they can do so turn out to be very different problems with very different remedies.

The first is that a regressor may be correlated with the error term. We saw three routes to it: a relevant variable left out of the model, a variable measured with error, and a variable that is determined jointly with the dependent one. In all three the orthogonality condition fails, and the consequence is the same and it is the serious one, the OLS estimators are no longer consistent. No sample size rescues them. The cure is to stop using the whole variation of the suspect regressor and to use only the part of it that is clean, which is what an instrument provides. From that single idea came the IV estimator, its two stage version when the instruments outnumber the regressors, and the battery of tests that must accompany them, Hausman for whether the problem exists at all, the first stage F for whether the instruments are strong enough to be worth using, and Sargan for whether they are exogenous, a test that is only available when the equation is over identified. We also paid the price of the method explicitly, the IV variance is never smaller than the OLS one, so an instrument should be reached for because the orthogonality condition fails, not out of caution.

Simultaneous equations were the same problem written as a system, and they forced us to be precise about identification. The reduced form is always estimable, the structural form is what we actually want, and the passage from one to the other is possible only when enough restrictions are imposed. The order condition counts them, the rank condition decides. Once an equation is identified, the estimation methods differ only in how much of the system they exploit, ILS and 2SLS and LIML equation by equation, 3SLS and FIML on the whole system at once, the second group being more efficient when the disturbances are correlated across equations and more fragile when any single equation is misspecified.

The second problem is of a completely different nature. Under multicollinearity the regressors are correlated with each other rather than with the error, nothing in the Gauss Markov theorem is violated, and the estimators remain unbiased and still have the smallest variance in their class. The trouble is that the smallest available variance is simply too large to be useful, the coefficients become unstable, the t ratios collapse while the \(R^2\) stays high, and signs turn against theory. Since the variance cannot be reduced without leaving the class of unbiased estimators, every remedy in the last part of the chapter does exactly that. Deleting a variable, shrinking the coefficients with ridge or lasso, or replacing the regressors by a smaller set of orthogonal combinations with principal components or partial least squares, all accept some bias in exchange for precision.

That trade is worth naming, because it is the pivot of the whole book. In the first half of this chapter bias was intolerable and we spent variance to remove it. In the second half bias became a price we were willing to pay. Which of the two is right depends on the question being asked, and that question, explaining versus predicting, is the one the machine learning chapter opens with.