Chapter 1 Introdcution

We invite you to join us in a guided tour of Econometrics and Machine Learning for Predictive Modeling. This book aims to be a one-stop shop for a rigorous introduction to fundamental concepts and their practical application. In the following chapters, we will cover a wide variety of models for predictive modelling, from the basic Simple Linear Model to more complex methods such as nonlinear and deep learning models.

In each chapter, we have given tremendous care to the exposition, which explains why the models work as they do even to readers with little to no mathematical background. We begin by discussing the mathematical foundations for each approach, which helps us put the theory on firm ground. Everywhere reasonable, we signal to the reader which work is considered keystone, so that writings of foundational nature can be identified easily.

Also, each section in this book concludes with a case study, coded in two programming languages – R and Python, which are two of the most popular programming languages. These case studies translate the ideas of each chapter into a real-world example.

The book will be structured into the following chapters:

Simple Linear Model

Here, we examine a simple approach to estimating outcomes based on one input factor. We’ll study how a response variable relates to a single predictor using linear regression. Through clear examples using R and Python, we’ll cover the key ideas of calculating coefficients, testing assumptions, and understanding the model’s implications. Some sentences will be concise, others more detailed, but all will aim for clarity. We will also discuss several estimation methods used in the context of linear models, each with its assumptions to ensure the validity of the model.

Multiple linear model

In the real world, we rarely or never see a variable dependent on just one input. The standard case is exemplified by the following example: Imagine you’re trying to understand why your car’s miles per gallon varies. You might consider factors such as the car’s weight, engine size, and even the terrain you’re driving on. This scenario resembles the Multiple Linear Model, which examines how multiple variables (like car weight and engine size) impact an outcome (such as miles per gallon). In this chapter, we’ll explore modeling situations where there are many potential “ingredients” influencing a result. We’ll also learn how to interpret the importance of different ingredients when multiple factors are at play.

Assumptions on the regressors

Each estimation method requires some assumptions, some of which are related to the predictors. In this chapter, we’ll cover these types of assumptions such as stochasticity, non-orthogonality, and multicollinearity. These ideas are important for building valid models. We’ll explain the concepts and provide examples. Following these rules helps to obtain reliable results from regression.

Assumptions on the errors

These types of assumptions are related to the errors of the theoretical linear model, which are the core ideas behind regression analysis. These errors, or residuals, come with some key assumptions we need to understand. We’ll explore normality (the errors follow a normal distribution), homoscedasticity (constant variance), and non-autocorrelation (errors aren’t linearly related to each other). If assumptions are broken, we’ll show you how to remedy the associated model.

Non linear models

Many real-world applications don’t follow a linear pattern. They exhibit a curved relationship. That’s where nonlinear models come in handy. There exist various methods like numerical optimization, quantile regression, and nonparametric models such as local estimation and splines, among others. With clear explanations and real examples, you’ll learn to spot and interpret those squiggly patterns lurking in your data.

Time series

Understanding patterns and trends over time is vital in many fields. Time Series Analysis is a specialized branch of statistics focused on sequentially collected data points. In Time Series Analysis, key ideas include values depending on prior ones, predictable cycles or trends, data fluctuating around a stable average, values correlating with their lags, and forecasting future values. This type of data requires different types of estimation methods. The common and traditional ones are ARIMA models that capture auto-regressive and moving average components, exponential smoothing data, state-space models incorporating unobserved values, and machine learning algorithms detecting complex patterns. All these methods and more will be uncovered in this chapter.

Machine learning

As data continues to grow exponentially, along with the increasing complexity of relationships, traditional classical methods often fall short in providing accurate predictions. This is where machine learning comes into play—a powerful tool for uncovering intricate patterns hidden within data. In this chapter, we will delve into the fundamental concepts of machine learning. We will explore commonly used algorithms such as decision trees, random forests, support vector machines, and others. Through practical examples, you will gain insight into how these robust models can effectively analyze data and generate precise predictions.

deep learning

In this chapter, we will walk through the fundamental principles of deep learning. We’ll dissect key concepts such as neural network architectures, activation functions, and optimization techniques. Along the way, we’ll explore renowned algorithms like Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs). Through practical illustrations and code snippets, you’ll uncover how these sophisticated models harness the power of data to drive insightful analysis and make precise predictions.

All mathematical results are demonstrated using their original papers whenever possible, and all the data used is either open-sourced or simulated.