Multivariate Regression extends simple linear regression to more than one independent variable (X), allowing complex relationships to be modeled where multiple factors influence a dependent variable (Y).Practical Example. Predicting the price of a house (Y) based on: Size (X₁), Number of bedrooms (X₂), and Location (X₃).The general equation is:\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_k X_k + \epsilon \]Where:- \( \beta_0 \): Intercept (value of Y when all X = 0).- \( \beta_1, \beta_2, \dots, \beta_k \): Coefficients of each variable.- \( \epsilon \): Random error. Expected value of zero for the optimal values of \(\beta\)How to Calculate the Coefficients?Ordinary Least Squares (OLS) Method. Minimizes the sum of the squared residuals:\[ \min \sum (Y_i - \hat{Y_i})^2 \]Step by Step:1. Build the Design Matrix (X):- Include a column of 1s for the intercept (\( \beta_0 \)).- Example for 3 variables:\[ X = \begin{bmatrix} 1 & X_{11} & X_{12} & X_{13} \\ 1 & X_{21} & X_{22} & X_{23} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & X_{n1} & X_{n2} & X_{n3} \end{bmatrix} \]2. Calculate the Coefficients:\[ \beta = (X^T X)^{-1} X^T Y \]- \( X^T \): Transposed matrix.- \( (X^T X)^{-1} \): Inverse matrix.Practical ExerciseData:Size (X₁)Bedrooms (X₂)Price (Y)502300603400702350Calculate:1. The multivariate regression equation.2. The R² and interpret it.Problems with Multiple Variables1. Multicollinearity: When Variables Get ConfusedMulticollinearity occurs when two or more independent variables (X) are highly correlated, making it difficult to distinguish their individual impact on the dependent variable (Y).Practical Example. Suppose we want to predict the price of a car (Y) using: Age of the car (X₁) and Mileage (X₂). There is a problem: age and mileage are strongly related (older cars tend to have more km). This inflates the variance of the coefficients, making the estimates unstable and difficult to interpret. Coefficients can become smaller and unstable, leading to wrong conclusions such as "coefficient X is not significant" when, in fact, it is important, but its contribution is being 'stolen' by another correlated variable ("Does mileage really impact the price, or is it just a side effect of age?").Solutions: Remove Redundant Variables (e.g., choose only "Age" or "Mileage"); to do this, create a correlation matrix between the variables and check those that have 80% or more correlation. Or transform the variables into normalized components using techniques such as Principal Component Analysis (PCA), where 2 variables can be explained by just 1 component.Below is a third column that represents the first two very well, so you can ignore the first two columns in your regression and use only the third:Height (cm)Weight (kg)PCA17065-0.5180801.516055-1.02. Overfitting: When the Model Memorizes the DataOverfitting occurs when the model fits the training data too much, capturing even the "noise" (random variations), and performing poorly on new data.Practical Example: Predicting student grades (Y) with: Study hours (X₁), Number of pencils (X₂), and Backpack color (X₃). Including "backpack color" (irrelevant) can make the model memorize nonexistent patterns, worsening generalization. When the model assigns importance to random factors (such as backpack color), it loses generalization capacity and begins to perform poorly on new data, since these invented "rules" do not repeat in reality. In addition, variables unrelated to the result dilute the impact of truly important variables, such as study hours. The model wastes energy trying to explain noise instead of focusing on the factors that really influence grades, making predictions less accurate and stable. Therefore, less is more: using only relevant variables ensures a simpler, more robust, and reliable model.Simple linear regression and multivariate regression with overfittingSolutions: Split the data into parts, train and test on different subsets, also called Cross-Validation (k-fold). Training R² = 0.99, test R² = 0.60 means Overfitting. Or remove non-significant variables (p-value > 0.05). Calculate the AIC or BIC to check whether so many variables are necessary.The AIC (Akaike Information Criterion) and the BIC (Bayesian Information Criterion) are statistical measures used to compare the goodness of fit of models, balancing model complexity (number of parameters) and its ability to explain the data. The AIC, developed by Hirotugu Akaike in 1974, is calculated by the formula \( AIC = 2k - 2\ln(L) \), where \( k \) is the number of parameters and \( L \) is the maximum likelihood of the model. The lower the AIC, the better the balance between fit and simplicity, penalizing overly complex models. It is widely used in model selection, especially when the goal is prediction, but it tends to favor more complex models compared to BIC in large samples. The BIC, also known as Schwarz Criterion, was proposed by Gideon Schwarz in 1978 and incorporates a greater penalty for the number of parameters, especially in large samples. Its formula is \( BIC = k\ln(n) - 2\ln(L) \), where \( n \) is the sample size. The BIC prioritizes simpler models and is stricter in parameter selection, often being preferred in contexts where the true structure of the model is unknown and the goal is to avoid overfitting.3. Missing Data and Extreme Values: When Data MisleadsMore variables means having a greater probability of having to deal with extreme values or missing data.Missing data can arise from non-response in surveys or technical failures, such as miscalibrated sensors. For example, in a medical study where 30% of patients omitted their income, this gap can bias analyses. To address this problem, there are two main approaches: deleting incomplete rows (which is only valid when the missing data is random) and imputation techniques, such as using mean/median for numerical data, regression for estimates based on other variables, or K-NN, which fills in values based on similar observations.Extreme values, such as a salary of R$5 million in a sample with an average of R$10 thousand, can completely distort the model's results. These extreme values are identifiable through boxplots (points beyond 1.5 times the interquartile range) or Z-scores (absolute values above 3). To minimize their effect, logarithmic or Box-Cox transformations are applied to smooth the disparity, Winsorization (replacement with the nearest boundary value), or, in justifiable cases, direct removal. Proper treatment preserves the model's robustness without losing critical information.Practical ExerciseSize (m²)BedroomsAge (years)Price (R$)5025300,00060310400,0007023350,00080415200,000Questions:1. Is there multicollinearity between "Size" and "Bedrooms"? Calculate the correlation.
Sign in and join the conversation
— Comments 0
, Reactions 1
Be the first to comment