6.12 Conclusion:

The word nonlinear covered four quite different departures from the classical model in this chapter, and it is worth separating them before moving on.

The first is the mildest. The model is still about the conditional mean and still estimated by least squares, but the parameters enter the function nonlinearly, so the normal equations can no longer be solved in closed form. Writing them with the pseudo regressors showed how close the structure stays to the linear case, and the Gauss-Newton algorithm did the rest by solving a sequence of linear problems. The price is that everything becomes iterative and everything becomes asymptotic. The exact distributions of the linear model are gone, and inference relies instead on the three large sample tests, Wald, likelihood ratio and Lagrange multiplier, which agree asymptotically and often disagree in the samples we actually have.

The second departure changes the criterion rather than the function. Least squares estimates a mean, and a mean is not always what we want, nor is it robust when the tail of the distribution is heavy. Quantile regression, and the least absolute deviation estimator as its median special case, replaces the squared loss with an asymmetric absolute one and estimates a quantile of the conditional distribution instead of its centre. That gives a picture of the whole distribution of \(y\) given \(x\), not just one point of it.

The third departure gives up the functional form altogether. Rather than assume a shape and estimate its parameters, the nonparametric estimators let the data choose the shape locally, from the crude binned estimator to the Nadaraya-Watson local constant, the local linear and local polynomial estimators, and finally the splines. What we saw there is that the choice of kernel matters very little and the choice of bandwidth matters enormously. The bandwidth is the real parameter of these methods, and choosing it is choosing a point on the trade off between a smooth curve that misses the structure and a wiggly one that follows the noise. The reader should keep this construction in mind, because it returns in the chapters ahead under other names.

The fourth departure is not about the shape of the relation at all but about the nature of the dependent variable. When \(y\) is a yes or no, a category, an ordered grade or a count, the linear model is wrong before any curvature is discussed, because its predictions leave the set of values \(y\) can take. The latent variable formulation gave us a way to think about it, a link function gave us a way to write it, and maximum likelihood replaced least squares as the estimation principle throughout. From there the models followed the structure of the data, logit and probit for two outcomes, the multinomial model for unordered categories, and for ordered ones the cumulative logit, the continuation ratio and the adjacent category models, each corresponding to a different way of cutting the order into comparisons.

The count models deserve a last word, because they are the clearest example in this book of a sequence of models built as answers to a failing assumption. The Poisson model imposes that the variance equals the mean. Real count data rarely obeys, so we tested the restriction, and each of the models that followed relaxes it in its own way, the quasi-Poisson by rescaling the variance, the negative binomial by adding a dispersion parameter, and the hurdle and zero inflated models by admitting that the zeros may not come from the same process as the positive counts. This is how applied modelling usually proceeds. One starts from the simplest specification, tests what it assumes, and lets the failure of the test dictate the next model rather than the other way round.

Finally, notice what quietly changed in this chapter. Maximum likelihood displaced least squares as the estimation principle, and it will not give the place back. It is the engine of the time series models of the next chapter, it reappears as the loss function of the machine learning chapter, and the link function that turns a linear index into a probability here is the same one that sits at the output of a neural network later in this book under the name softmax.