From Econometrics to Deep Learning with R and Python
1
Introdcution
2
Simple Linear Regression
2.1
General concepts
2.1.1
Random model
2.1.2
Model variables
2.1.3
Estimation
2.1.4
Comparison of estimation methods
2.2
Estimation methods
2.2.1
Ordinary Least Squares (OLS)
2.2.2
Maximum likelihood method ML
2.2.3
Method of moments MM
2.2.4
The best linear unbiased estimator BLUE.
2.3
Parameter distributions
2.3.1
Probability distributions of variables and parameters in terms of the (unknown) error variance
2.3.2
Probability distributions of variables and parameters in terms of the (known) residual variance
2.4
Hypothesis testing
2.4.1
Test the significance of estimates:
2.4.2
Confidence intervals:
2.4.3
test the overall significance of the model
2.5
Explanation and prediction
2.5.1
Explanation:
2.5.2
Prediction:
2.6
Regression with transformed variables
2.6.1
Centered variables
2.6.2
Polynomial transformation:
2.6.3
The Logarithmic transformation:
2.6.4
The inverse hyperbolic sine transformation
2.7
Regression with categorical varaible
2.8
Application
3
Multiple Linear Regression
3.1
The true model
3.2
Esimation by OLS method
3.2.1
The estimated model:
3.2.2
The means of the OLS estimators
3.2.3
The variance matrix of the OLS estimators
3.2.4
Estimation of the error variance
3.2.5
The variance analysis equation
3.3
Estimation by maximum likelihood method ML
3.4
Estimation by method of moments MM
3.5
Estimation by Gauss-markov method
3.6
Assumptions
3.7
Probability distributions of variables and parameters
3.7.1
Probability distribution in terms of the unknown
\(\sigma^2\)
3.7.2
Probability distribution in terms of the estimated variance
\(s^2\)
3.8
Selection of regressors
3.8.1
Including all the DGP regressors
3.8.2
Omitting some of the DGP regressors
3.8.3
Including irrelevant regressors
3.9
Tests and hypothese
3.9.1
The coefficient of determination
\(R^2\)
3.9.2
Hypothesis Formulation
3.9.3
Restricted model
3.9.4
Wald test
3.9.5
LM-test
3.9.6
LR-test
3.9.7
W vs LM vs LR
3.10
Regression with categorical variables
3.10.1
Regression with one categorical variables
3.10.2
Regression with two categorical variables
3.10.3
Regression with mix of categorical and continuous variables
3.11
Regression with large number of regressors
3.11.1
All possible regression
3.11.2
compare two specific models
3.11.3
Forward selection
3.11.4
Backawrd elimination
3.11.5
Stepwise selection
3.12
Conclusion
4
Assumptions on the regressors
4.1
Stochastic regressors
4.1.1
Statistical properties of the estimators
4.1.2
Asumptotic analysis and consistency
4.2
Non orthogonality condition
4.2.1
Omission of relevant variables
4.2.2
Measurement errors in the regressors
4.2.3
Simultaneous equations
4.3
Multicolinearity:
4.3.1
Multicollinearity detection
4.3.2
Mylticollinearity remedies
4.4
Conclusion
5
Assumptions on the disturbances
5.1
Autoccorrelation
5.1.1
The OLS properties under autocorrelation
5.1.2
Causes of autocarrelation
5.1.3
Testing for autocorrelation
5.1.4
Estimation under autocorrelation
5.2
Heteroskedasticity
5.2.1
The OLS properties under hetroskedasticity
5.2.2
Causes of heteroskedasticity
5.2.3
Testing for heteroskedasticity
5.2.4
Estimation under heteroskedasticity
5.3
Non normality
5.3.1
Testing the normality assumption
5.3.2
Bootstrapped confidence intervals
6
Non linear models
6.1
Non liner least squares regression NLS
6.1.1
Gauss-Newton method
6.1.2
Consistency of the NLS estimators
6.1.3
Hypothesis testing
6.2
Quantile regression
6.3
Non parametric regressions
6.3.1
Local Constant estimator NW
6.3.2
Local linear estimator LL
6.3.3
Local Polynomial Estimator LP
6.3.4
Splines
6.4
Binary response variable
6.5
Multinomial response variable
6.6
Ordinal response variable
6.6.1
Cumulative logit model
6.6.2
Continuation ratio model
6.6.3
Adjacent category logistic model
6.7
Count data model
6.7.1
Poisson model
6.7.2
goodness of fit
6.7.3
Overdisperssion
6.8
The quasi-Poisson model
6.9
The negative binomial model
6.10
The Hurdle model
6.11
The Zero inflated model
6.12
Conclusion:
7
Time series
7.1
Concepts of Time Series
7.1.1
Components of Times Series
7.1.2
Stochactic process
7.1.3
White noise
7.1.4
Random walk
7.1.5
Autoregressive process AR
7.1.6
Moving average process MA
7.1.7
ARMA process
7.1.8
Autocorrelation function ACF
7.1.9
Partiall autocorrelation function PACF
7.2
Stationarity
7.2.1
Unit root
7.2.2
Autocorrelation function ACF
7.2.3
Partiall autocorrelation function PACF
7.3
ARIMA models
7.3.1
AR model
7.3.2
Impulse response function IRF
7.3.3
MA model
7.3.4
ARMA model
7.3.5
Integrated ARMA model ARIMA
7.3.6
ARIMAX model
7.3.7
SARIMA model
7.4
ARCH AND GARCH MODELS
7.4.1
ARCH model
7.4.2
GARCH model
7.4.3
TGARCH MODEL
7.5
Cointegration
7.5.1
The error correction model ECM
7.6
Autoregressive distributed models ARDL
7.7
Multivariate time series
7.7.1
VAR models
7.7.2
VECM models
8
Machine Learning
8.1
Concepts and definitions
8.1.1
Supervised learning: Regression and Classification
8.1.2
Multivariate regression
8.1.3
multi-label classfication
8.1.4
Unsupervied learning: Clustring
8.1.5
Semi supervised learning
8.1.6
The loss function and decision function
8.1.7
Training and Testing
8.1.8
underfiting and overfiting problem
8.1.9
Metrics and scores
8.2
Supervided learning models
8.2.1
Linear regression
8.2.2
Ridge regresson
8.2.3
Lasso regression
8.2.4
Elastic net regression
8.2.5
Linear discriminant analysis
8.2.6
Quadratic discriminant analysis
8.2.7
Support vector machine
8.2.8
Naive bayes model
8.2.9
K-nearest neighors
8.2.10
stochastic gradient descent
8.2.11
Decision trees
8.2.12
Random foreset
8.2.13
ensemble methods
8.3
Unsupervised learning
8.3.1
Gaussian mixture models
8.3.2
K-means
8.3.3
K-mediods
8.3.4
Hierarchical clustering
8.3.5
DBSCAN
8.3.6
PCA method
8.3.7
ICA method
8.3.8
Factor analysis
8.3.9
Latend dirichlet allocation
8.3.10
Manifold learning
8.4
Imputing missing values
8.5
Models for outliers detection
8.6
Pre-processing data
9
Deep learning
9.1
From the linear model to the network
9.1.1
The artificial neuron
9.1.2
The limits of a single neuron
9.1.3
The multilayer perceptron
9.1.4
The universal approximation theorem
9.2
Activation functions
9.2.1
Why a non linearity is indispensable
9.2.2
The sigmoid and the hyperbolic tangent
9.2.3
ReLU and its variants
9.2.4
The output layer and the softmax
9.3
Training a network
9.3.1
Forward propagation
9.3.2
The loss function
9.3.3
Backpropagation
9.3.4
Gradient descent and its variants
9.3.5
The learning rate
9.3.6
Batch, epoch and mini-batch
9.3.7
Vanishing and exploding gradients
9.3.8
The initialization of the weights
9.4
Regularization
9.4.1
Overfitting in a network
9.4.2
Weight decay
9.4.3
Dropout
9.4.4
Early stopping
9.4.5
Batch normalization
9.4.6
Data augmentation
9.5
Convolutional neural networks
9.5.1
Why convolution
9.5.2
The convolution operation
9.5.3
Filters, feature maps and channels
9.5.4
Pooling
9.5.5
A complete architecture
9.5.6
Transfer learning
9.6
Recurrent neural networks
9.6.1
Sequential data
9.6.2
The recurrent cell
9.6.3
Backpropagation through time
9.6.4
LSTM and GRU
9.6.5
Application to a time series
9.7
Generative adversarial networks
9.7.1
The adversarial idea
9.7.2
Generator and discriminator
9.7.3
The minimax objective
9.7.4
Why the training is difficult
9.7.5
A worked example
9.8
A practical guide
9.8.1
Choosing an architecture
9.8.2
Reading a learning curve
9.8.3
The most common mistakes
10
Final Words
10.1
Two centuries in one book
10.1.1
From least squares to the probability approach
10.1.2
The econometric synthesis
10.1.3
The learning turn
10.1.4
The deep turn
10.1.5
The chronology in one picture
10.2
The thread that runs through the book
10.2.1
Two cultures, one trade-off
10.2.2
What the change of culture really was
10.2.3
What has not changed at all
10.3
What comes next
10.3.1
The two cultures are converging
10.3.2
Architectures are becoming general
10.3.3
Interpretability stops being optional
10.3.4
The reproducibility problem
10.3.5
What will not be solved
10.4
Using this book
References
About the author
Dr. Abdelkader Metales
From Econometrics to Deep Learning with R and Python
3.12
Conclusion
In this chapter, we learned how to estimate a linear model using OLS methods.