Multivariate regression (also called multiple linear regression) extends simple linear regression to more than one independent variable (X), allowing us to model situations in which several factors influence the dependent variable (Y).Example: predicting the price of a house (Y) based on its size (X₁), number of bedrooms (X₂) and location (X₃). Qualitative variables such as location enter the model as indicator (dummy) variables (0 or 1), one for each neighbourhood minus one.The general equation is:\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_k X_k + \epsilon \]Where:\( \beta_0 \): intercept (predicted value of Y when all X are 0);\( \beta_1, \beta_2, \dots, \beta_k \): coefficients of each variable. Each one measures the expected change in Y when that X increases by 1 unit and the others are held constant;\( \epsilon \): random error, which we assume has a mean of zero.How to Calculate the Coefficients?We use the ordinary least squares (OLS) method, which chooses the coefficients that minimize the sum of squared residuals:\[ \min \sum (Y_i - \hat{Y}_i)^2 \]Step by step:Build the design matrix (X):include a column of 1s for the intercept (\( \beta_0 \));example for 3 variables and \( n \) observations: \[ X = \begin{bmatrix} 1 & X_{11} & X_{12} & X_{13} \\ 1 & X_{21} & X_{22} & X_{23} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & X_{n1} & X_{n2} & X_{n3} \end{bmatrix} \]Calculate the coefficients: \[ \hat{\beta} = (X^T X)^{-1} X^T Y \]\( X^T \): transpose of X;\( (X^T X)^{-1} \): inverse of the product \( X^T X \).For the inverse to exist, no column of X can be an exact combination of the others, and there must be more observations than coefficients: with \( k \) variables, the model estimates \( k + 1 \) coefficients.A multivariate regression uses 50 observations and 3 independent variables, with an intercept. What is the dimension of the design matrix X?Answer: 50 × 4 — one row per observation and one column per variable, plus the column of 1s for the intercept. The 50 × 3 option forgets that column; 4 × 4 is the dimension of X transpose times X.Practical ExerciseData:Size (X₁)Bedrooms (X₂)Price (Y)502300603400702350Calculate:The multivariate regression equation.The R², and interpret it.Answer: with 3 observations and 3 coefficients (\( \beta_0, \beta_1, \beta_2 \)), the system has an exact solution. Subtracting row 1 from row 3: \( 50 = 20\beta_1 \), so \( \beta_1 = 2.5 \). Subtracting row 1 from row 2: \( 100 = 10\beta_1 + \beta_2 \), so \( \beta_2 = 75 \). Finally, \( \beta_0 = 300 - 50 \times 2.5 - 2 \times 75 = 25 \):\[ \hat{Y} = 25 + 2.5X_1 + 75X_2 \]The R² is exactly 1, because the equation passes through every point. But that does not mean the model is perfect: with as many coefficients as observations, an exact fit is always possible, no degrees of freedom are left to measure the error, and nothing guarantees that the model will get a new house right. This is the extreme case of overfitting, which we will see next.Problems with Multiple Variables1. Multicollinearity: When Variables Get Mixed UpMulticollinearity occurs when two or more independent variables (X) are highly correlated, which makes it hard to separate the effect of each one on the dependent variable (Y).Example: suppose we want to predict the price of a car (Y) using the age of the car (X₁) and its mileage (X₂). The problem is that age and mileage go together (older cars tend to have more miles). This inflates the variance (the standard errors) of the coefficients, making the estimates unstable and hard to interpret: the coefficients change a lot when a few data points are added or removed, they may even flip sign, and the t-test may conclude that a variable is not significant when it is actually important but its contribution is being "stolen" by the other one. The question "does mileage really affect the price, or is it just an effect of age?" is left unanswered.How to detect it: build a correlation matrix of the independent variables and look at pairs with a correlation of 0.8 or more (in absolute value). A more complete measure is the VIF (variance inflation factor) of each variable:\[ VIF_j = \frac{1}{1 - R_j^2} \]where \( R_j^2 \) is the R² of the regression of variable \( X_j \) on the other independent variables. Values above 5, or above 10 under more lenient criteria, indicate worrying multicollinearity.Solutions: remove redundant variables (for example, keep only age or only mileage), or combine the correlated variables into components with Principal Component Analysis (PCA), so that two variables can be summarized by a single component.In the table below, height and weight have a correlation of 0.99. The third column is the first principal component of the two standardized variables, and on its own it summarizes 99.7% of the variation of both. So you can leave the first two columns out of the regression and use only the third:Height (cm)Weight (kg)PCA (1st component)17065−0.09180801.4616055−1.36The price of this solution is interpretation: the component's coefficient no longer tells you the effect of 1 cm or 1 kg, but the effect of "being bigger" on both measures at once.In a car price model, age and mileage have a correlation of 0.95. What should you expect when including both in the regression?Answer: Inflated standard errors and unstable coefficients, which may even appear non-significant — the two variables carry almost the same information, and the model cannot separate the effect of each one. The model can still be estimated (that would only fail with a correlation of exactly 1), and R² does not decrease when a variable is added.Regressing variable X1 on the other independent variables gave R² = 0.90. What is the VIF of X1?Answer: 10 — VIF = 1/(1 minus 0.90) = 1/0.10 = 10, the threshold at which most criteria already consider multicollinearity severe. The 1.11 option is 1/0.90, which uses R² instead of 1 minus R².2. Overfitting: When the Model Memorizes the DataOverfitting occurs when the model fits the training data too closely, capturing even the noise (random variation), and performs poorly on new data.Example: predicting students' scores (Y) with study hours (X₁), number of pencils (X₂) and backpack colour (X₃). Including the backpack colour, which is irrelevant, can make the model memorize patterns that do not exist. When the model gives weight to random factors, it loses its ability to generalize, since those invented "rules" do not repeat in reality. In addition, variables unrelated to the outcome use up degrees of freedom and increase the uncertainty of the estimates for the variables that really matter, such as study hours. That is why less is more: using only relevant variables produces a simpler, more robust and more reliable model.Note that the ordinary R² never decreases when a variable is added, even an irrelevant one. So, to compare models with different numbers of variables, use the adjusted R², which accounts for the number of variables.Simple linear regression and multivariate regression with overfittingSolutions:Cross-validation (k-fold): split the data into parts, train the model on some and test it on the others. If R² is 0.99 on training and 0.60 on testing, there is overfitting.Remove non-significant variables (p-value > 0.05), one at a time, re-evaluating the model.Use information criteria (AIC or BIC) to check whether so many variables are really needed.The AIC (Akaike information criterion) and the BIC (Bayesian information criterion) compare models by balancing goodness of fit against complexity (the number of parameters). The AIC was proposed by Hirotugu Akaike in 1974 and is calculated as \( AIC = 2k - 2\ln(L) \), where \( k \) is the number of parameters and \( L \) is the model's maximum likelihood. The lower the AIC, the better the balance between fit and simplicity. The BIC, also called the Schwarz criterion, was proposed by Gideon Schwarz in 1978: \( BIC = k\ln(n) - 2\ln(L) \), where \( n \) is the sample size. Since \( \ln(n) \) exceeds 2 from \( n = 8 \) onwards, the BIC penalizes each parameter more than the AIC in most samples and tends to choose simpler models. In practice, the AIC is usually preferred when the goal is to predict well, and the BIC when the goal is to identify which variables really belong in the model.When testing a model, you get R² = 0.98 on the training data and R² = 0.55 on the test data. What is the most likely diagnosis?Answer: Overfitting: the model memorized the noise in the training data — a large drop from training to testing is the classic sign. A model that is too simple would do poorly on both sets, not just on the test set.With n = 100, model A has k = 3 and ln L = −120; model B has k = 6 and ln L = −116. What do the AIC and the BIC say?Answer: The AIC prefers B and the BIC prefers A — AIC: A = 6 + 240 = 246 and B = 12 + 232 = 244 (B lower). BIC: A = 3 × 4.61 + 240 ≈ 253.8 and B = 6 × 4.61 + 232 ≈ 259.6 (A lower). The BIC's larger penalty per parameter makes the difference.3. Missing Data and Extreme Values: When the Data MisleadThe more variables, the greater the chance of having to deal with extreme values or missing data.Missing data can arise from non-response in surveys or from technical failures, such as miscalibrated sensors. For example, in a medical study in which 30% of patients did not report their income, that gap can bias the analysis. There are two main approaches: deleting incomplete rows (which is only safe when data are missing completely at random) or using imputation techniques, such as filling in the mean or median, estimating the value by regression from the other variables, or using K-NN, which fills in values based on the most similar observations.Extreme values, such as a salary of R$ 5 million (Brazilian reais) in a sample with an average of R$ 10 thousand, can completely distort the model's results. They can be identified with boxplots (points more than 1.5 times the interquartile range below the 1st quartile or above the 3rd) or with Z-scores (absolute values above 3). To reduce their effect, we apply logarithmic or Box-Cox transformations, which reduce the disparity; winsorization (replacing values beyond a percentile, such as 1% or 99%, with the value of that percentile); or, in justified cases, removal. Proper treatment keeps the model robust without losing important information.Practical Exercise: MulticollinearitySize (m²)BedroomsAge (years)Price (R$)5025300,00060310400,0007023350,00080415200,000Questions:Is there multicollinearity between "Size" and "Bedrooms"? Calculate the correlation.And between "Bedrooms" and "Age"?Is it possible to estimate a reliable regression of price on the three variables using only these 4 houses?Answers:The correlation between size and bedrooms is \( r \approx 0.67 \): moderate, below the 0.8 threshold. By the correlation-matrix rule, it is not severe multicollinearity.The correlation between bedrooms and age is \( r \approx 0.99 \): very strong multicollinearity. In this sample, the houses with more bedrooms are precisely the oldest ones, and the model would have no way of separating the two effects.No. There are 4 observations for 4 coefficients (intercept and three variables): the fit is exact (R² = 1) and the coefficients come out absurd, with each extra bedroom worth about R$ 2 million and each extra square metre reducing the price. This is multicollinearity and overfitting acting together: many more houses would be needed, and probably dropping either the number of bedrooms or the age.
ردود الفعل
0
0 تعليقات
كن أول من يعلق
لا شيء هنا بعد.