9.8 A practical guide

This last section gathers what is not in the equations but decides the outcome.

9.8.1 Choosing an architecture

The first rule is the least fashionable: do not start with a network. On tabular data, which is the ordinary material of econometrics, gradient boosting of the previous chapter is very often better, faster to train and easier to explain. Networks dominate where the data have a structure that they can exploit, images, sound, text, long sequences, and where the sample is large.

If a network is justified, the architecture follows the structure of the data. A grid calls for convolutions, a sequence for a recurrent or attention based model, a set of unordered features for dense layers. The depth and the width are then chosen by starting small and growing only while the validation error improves.

Finally, the order in which the hyperparameters deserve attention is roughly: the learning rate first, by a wide margin, then the architecture, then the regularization, and only afterwards the rest. Spending a day on the choice of the optimizer before having settled the learning rate is a common way of wasting it.

9.8.2 Reading a learning curve

The two curves, training and validation, diagnose most situations at a glance:

• both high and flat: underfitting. The model is too small, or the learning rate is wrong, or the training is too short.

• training low, validation much higher and rising: overfitting. More data, more regularization, or early stopping.

• the loss oscillates violently or becomes NaN: the learning rate is too large, or the gradients explode. Reduce it, clip the gradients, check for a division by zero or a logarithm of zero in a custom loss.

• the loss does not move at all from the first epoch: something is disconnected. A frozen layer, a learning rate of zero, a forgotten zero_grad, or labels that do not correspond to the inputs.

• the validation curve is below the training curve: not an error in general, since regularizers such as dropout are active during training and not during evaluation.

A last check, cheap and very informative: make the model overfit a handful of observations on purpose. A correct implementation must reach a loss close to zero on twenty examples. If it cannot, the problem is in the code, not in the data or the capacity, and no amount of tuning will fix it.

9.8.3 The most common mistakes

Ranked roughly by how often they occur:

1. Leakage between the training and the test set. Scaling, imputing or selecting variables before the split, as the previous chapter insisted. On sequences, taking a random split instead of a chronological one, which lets the model read the future.

2. Forgetting the evaluation mode. Dropout and batch normalization behave differently in training and in evaluation. A model evaluated in training mode gives noisy and pessimistic results.

3. Unscaled inputs. A network with inputs of order one thousand produces enormous gradients at the first layer and learns nothing.

4. An inappropriate metric. Accuracy on an unbalanced problem, exactly as in the previous chapter.

5. A learning rate chosen without looking. It is the parameter that matters most and it costs a few minutes to explore.

6. Judging on the test set repeatedly. Looking at it, changing the model, looking again turns it into a training set. Keep a validation set for the decisions and the test set for the single final measurement.

7. Not fixing the seeds. A network is initialized at random, the batches are shuffled at random, and the dropout is random. Without fixed seeds an experiment is not reproducible, and two architectures cannot be compared honestly.

With this the chapter, and the tour begun with the simple linear model, comes to an end. The path went from a model of two parameters whose every property could be proved, to models of several million whose behaviour is established by experiment. The objective changed along the way, from explaining to predicting, and it is worth ending on the fact that the two questions remain distinct: a network that predicts remarkably well says nothing about why, and the econometric tools of the first chapters remain the ones to reach for when the question is one of effect rather than of forecast.