Section 2.11 Python Code and Analysis¶
Birthweight Data Set Example¶
The birthweight.csv data set contains data from a study on the birth weight of 42 babies. The variables in the data set are the following:
ID: Unique Identification number of a babyLength: Length of the baby at time of birth in cm ($X_1$)Birthweight: Weight of the baby at time of birth in kg ($Y$)Headcirc: Head circumference of the baby at time of birth in cm ($X_2$)Gestation: Completed weeks of gestation ($X_3$)smoker: Mother is/is not a smoker {0: No, 1: Yes} ($X_4$)mage: Mother's age at time of birth ($X_5$)mnocig: Mother's number of cigarettes smoked per day ($X_6$)mheight: Mother's height in cm ($X_7$)mppwt: Mother's pre-pregnancy weight ($X_8$)fage: Father's age at time of birth ($X_9$)fedyrs: Father's years of education ($X_{10}$)fnocig: Father's number of cigarettes smoked per day ($X_{11}$)fheight: Father's height ($X_{12}$)
## Import necessary libraries
import pandas as pd
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import seaborn as sns
import statsmodels.api as sm
import statsmodels.formula.api as smf
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.graphics.gofplots import ProbPlot
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
sns.set()
# load data
birthweight = pd.read_csv("Birthweight.csv")
print("Shape:", birthweight.shape)
birthweight.head()
Shape: (42, 14)
| ID | Length | Birthweight | Headcirc | Gestation | smoker | mage | mnocig | mheight | mppwt | fage | fedyrs | fnocig | fheight | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1360 | 56 | 4.55 | 34 | 44 | 0 | 20 | 0 | 162 | 57 | 23 | 10 | 35 | 179 |
| 1 | 1016 | 53 | 4.32 | 36 | 40 | 0 | 19 | 0 | 171 | 62 | 19 | 12 | 0 | 183 |
| 2 | 462 | 58 | 4.10 | 39 | 41 | 0 | 35 | 0 | 172 | 58 | 31 | 16 | 25 | 185 |
| 3 | 1187 | 53 | 4.07 | 38 | 44 | 0 | 20 | 0 | 174 | 68 | 26 | 14 | 25 | 189 |
| 4 | 553 | 54 | 3.94 | 37 | 42 | 0 | 24 | 0 | 175 | 66 | 30 | 12 | 0 | 184 |
The ID variable is not of interest, so we remove it from the data frame:
birthweight = birthweight.drop('ID', axis=1)
Fitting a Multiple Linear Regression¶
We begin by fitting a linear regression model that includes all predictors and examine the least-squares summary:
# statsmodels support R-style formula but not the period operator
# so we must construct the full formula manually
var_names = birthweight.columns.drop(['Birthweight'])
fml_full = 'Birthweight ~ ' + ' + '.join(x for x in var_names)
birthweight_mlr_full = smf.ols(formula=fml_full, data=birthweight).fit()
birthweight_mlr_full.summary()
| Dep. Variable: | Birthweight | R-squared: | 0.783 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.694 |
| Method: | Least Squares | F-statistic: | 8.742 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 1.01e-06 |
| Time: | 01:31:58 | Log-Likelihood: | -5.7803 |
| No. Observations: | 42 | AIC: | 37.56 |
| Df Residuals: | 29 | BIC: | 60.15 |
| Df Model: | 12 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | -3.8123 | 2.080 | -1.833 | 0.077 | -8.067 | 0.442 |
| Length | 0.0342 | 0.031 | 1.086 | 0.287 | -0.030 | 0.098 |
| Headcirc | 0.0866 | 0.029 | 2.937 | 0.006 | 0.026 | 0.147 |
| Gestation | 0.0966 | 0.031 | 3.083 | 0.004 | 0.033 | 0.161 |
| smoker | -0.2439 | 0.169 | -1.447 | 0.159 | -0.589 | 0.101 |
| mage | -0.0190 | 0.019 | -1.025 | 0.314 | -0.057 | 0.019 |
| mnocig | 0.0011 | 0.007 | 0.155 | 0.878 | -0.013 | 0.015 |
| mheight | 0.0042 | 0.014 | 0.296 | 0.769 | -0.025 | 0.033 |
| mppwt | 0.0103 | 0.011 | 0.921 | 0.364 | -0.013 | 0.033 |
| fage | 0.0072 | 0.016 | 0.440 | 0.663 | -0.026 | 0.041 |
| fedyrs | 0.0032 | 0.031 | 0.104 | 0.918 | -0.060 | 0.066 |
| fnocig | 0.0034 | 0.004 | 0.858 | 0.398 | -0.005 | 0.012 |
| fheight | -0.0133 | 0.010 | -1.383 | 0.177 | -0.033 | 0.006 |
| Omnibus: | 5.590 | Durbin-Watson: | 1.680 |
|---|---|---|---|
| Prob(Omnibus): | 0.061 | Jarque-Bera (JB): | 3.108 |
| Skew: | 0.448 | Prob(JB): | 0.211 |
| Kurtosis: | 2.013 | Cond. No. | 1.07e+04 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.07e+04. This might indicate that there are
strong multicollinearity or other numerical problems.
The objects returned by smf.ols can be inspected with:
[x for x in dir(birthweight_mlr_full) if not x.startswith('_')]
['HC0_se', 'HC1_se', 'HC2_se', 'HC3_se', 'aic', 'bic', 'bse', 'centered_tss', 'compare_f_test', 'compare_lm_test', 'compare_lr_test', 'condition_number', 'conf_int', 'conf_int_el', 'cov_HC0', 'cov_HC1', 'cov_HC2', 'cov_HC3', 'cov_kwds', 'cov_params', 'cov_type', 'df_model', 'df_resid', 'diagn', 'eigenvals', 'el_test', 'ess', 'f_pvalue', 'f_test', 'fittedvalues', 'fvalue', 'get_influence', 'get_prediction', 'get_robustcov_results', 'info_criteria', 'initialize', 'k_constant', 'llf', 'load', 'model', 'mse_model', 'mse_resid', 'mse_total', 'nobs', 'normalized_cov_params', 'outlier_test', 'params', 'predict', 'pvalues', 'remove_data', 'resid', 'resid_pearson', 'rsquared', 'rsquared_adj', 'save', 'scale', 'ssr', 'summary', 'summary2', 't_test', 't_test_pairwise', 'tvalues', 'uncentered_tss', 'use_t', 'wald_test', 'wald_test_terms', 'wresid']
The design matrix is obtained by
# The design matrix (including intercept) is stored in the .model.exog attribute
design_birthweight = birthweight_mlr_full.model.exog
The estimated regression coefficients (including the intercept) are
# The coefficients are stored in the .params attribute
birthweight_mlr_full.params
Intercept -3.812311 Length 0.034152 Headcirc 0.086594 Gestation 0.096565 smoker -0.243895 mage -0.019004 mnocig 0.001064 mheight 0.004187 mppwt 0.010346 fage 0.007187 fedyrs 0.003220 fnocig 0.003445 fheight -0.013293 dtype: float64
Fitted values and residuals are extracted as
full_model_residuals = birthweight_mlr_full.resid
full_model_fitted = birthweight_mlr_full.fittedvalues
Residual standard error and R2¶
The residual variance estimator is
$$\hat{\sigma}^2 = \frac{1}{n-p-1}\sum_i r_i^2 = \frac{\text{RSS}}{n-p-1}$$
To obtain an estimator for $\hat{\sigma}$, we can either use the formula (from the notes):
n = birthweight.shape[0]
p = birthweight.shape[1] - 1 # Number of predictors (excluding response)
rse = np.sqrt(np.sum(full_model_residuals ** 2) / (n - p - 1))
print(f"Manual RSE calculation: {rse:.6f}")
Manual RSE calculation: 0.334163
or we can extract it from the ols object:
# The model's 'scale' attribute is the estimated error variance (sigma-squared)
sigma_from_model = np.sqrt(birthweight_mlr_full.scale)
print(f"RSE from model (sqrt of scale): {sigma_from_model:.6f}")
RSE from model (sqrt of scale): 0.334163
The coefficient of determination computes as: $$ \frac{\sum_i (\hat{y}_i - \bar{y})^2}{\sum_i (y_i - \bar{y})^2} = 1 - \frac{\text{RSS}}{\text{TSS}}$$ We can extract $R^2$ from the output or compute it:
print(f"R-squared from model: {birthweight_mlr_full.rsquared:.6f}")
rss = np.sum(full_model_residuals ** 2)
tss = np.sum((birthweight['Birthweight'] -
birthweight['Birthweight'].mean()) ** 2)
r_squared_manual1 = 1 - (rss / tss)
r_squared_manual2 = 1 - np.var(full_model_residuals) / \
np.var(birthweight['Birthweight'])
print(
f"Manual R-squared calculations: {r_squared_manual1:.6f} and {r_squared_manual2:.6f}")
R-squared from model: 0.783425 Manual R-squared calculations: 0.783425 and 0.783425
Invariance of $R^2$ & Fitted Values¶
We verify the invariance property of 𝑅2 by centering one of the predictors:
# Center the Length variable
birthweight_new = birthweight.copy()
birthweight_new['Length_centered'] = birthweight_new['Length'] - \
birthweight_new['Length'].mean()
# Model with original `Length` (same as birthweight_mlr_full)
model_orig = smf.ols(fml_full, data=birthweight).fit()
# Model with centered `Length`
# We create a formula that includes all columns from birthweight_new except 'Length'
predictors_centered = birthweight_new.columns.drop(['Birthweight', 'Length'])
fml_centered = 'Birthweight ~ ' + ' + '.join(predictors_centered)
model_centered = smf.ols(fml_centered, data=birthweight_new).fit()
print(f"R-squared with original `Length`: {model_orig.rsquared:.6f}")
print(f"R-squared with centered `Length`: {model_centered.rsquared:.6f}")
R-squared with original `Length`: 0.783425 R-squared with centered `Length`: 0.783425
The fitted values from the two models are essentially identical (any tiny difference is due to floating-point rounding):
fit_diff = np.sum(model_orig.fittedvalues - model_centered.fittedvalues)
print(f"Sum of differences in fitted values: {fit_diff}")
Sum of differences in fitted values: 6.932232565759477e-13
$R^2$ and $\text{RSS}$ behavior when adding more $X$'s¶
We can also visualize how the $R^2$ and $\text{RSS}$ change as a function of the number of variables in the model to observe that adding a new predictor, even if it is randomly generated and has nothing to do with $Y$ will decrease $\text{RSS}$ and therefore increase $R^2$.
So, start with the null model that contains only the intercept and then we add variables one at a time, keeping the $R^2$ and $\text{RSS}$ that we will plot in the end. We will also randomly generate a new variable randomvar to illustrate that even an unrelevant variable has an effect on both $R^2$ and $\text{RSS}$
:
birthweight_new = birthweight.copy()
birthweight_new['randomvar'] = np.random.normal(size=birthweight_new.shape[0])
predictors = birthweight_new.columns.drop(['Birthweight']).tolist()
rsq = []
rss = []
# Model with only intercept
intercept_model = smf.ols('Birthweight ~ 1', data=birthweight_new).fit()
rsq.append(intercept_model.rsquared)
rss.append(intercept_model.ssr) # ssr is Sum of Squared Residuals
# Sequentially add predictors
for i in range(len(predictors)):
current_predictors = predictors[:(i + 1)]
formula = 'Birthweight ~ ' + ' + '.join(current_predictors)
model = smf.ols(formula, data=birthweight_new).fit()
rsq.append(model.rsquared)
rss.append(model.ssr)
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))
sns.lineplot(x=range(len(rsq)), y=rsq, lw=2, color='orange', marker='o', ax=ax1)
ax1.set_xlabel("No. of Predictors")
ax1.set_ylabel("R-square")
ax1.set_title("R-squared vs. Number of Predictors")
sns.lineplot(x=range(len(rss)), y=rss, lw=2, color='blue', marker='o', ax=ax2)
ax2.set_xlabel("No. of Predictors")
ax2.set_ylabel("RSS")
ax2.set_title("RSS vs. Number of Predictors")
plt.show()
This is to illustrate why $R^2$ and $\text{RSS}$ are not good metrics when it comes to model selection. They are useful, because they helps us understand the quality of the model or $\text{RSS}$ is a key ingredient in constructing hypothesis tests, but they are never used for model selection.
Rank Deficiency¶
We have rank deficiency when $\mathbf{X}$ is not of full rank which leads to the $(\mathbf{X}^\top\mathbf{X})$ matrix not being invertible.
Here we illustrate a scenario where it is easy to identify the cause of the problem. (The error message you see below the code is because of that.)
# Create a new variable: the average height of both parents
birthweight_new = birthweight.copy()
birthweight_new['aveheight'] = (
birthweight_new['mheight'] + birthweight_new['fheight']) / 2
var_names = birthweight_new.columns.drop(['Birthweight'])
fml_full_new = "Birthweight" + "~" + "+".join(x for x in var_names)
# Regress birthweight against all predictors
model_rankdef = smf.ols(fml_full_new, data=birthweight_new).fit(method='qr')
model_rankdef.summary()
/Users/alexandrachronopoulou/psl-book-env/lib/python3.11/site-packages/statsmodels/regression/linear_model.py:1884: RuntimeWarning: invalid value encountered in sqrt return np.sqrt(np.diag(self.cov_params()))
| Dep. Variable: | Birthweight | R-squared: | 0.740 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.632 |
| Method: | Least Squares | F-statistic: | 6.876 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 1.11e-05 |
| Time: | 01:31:58 | Log-Likelihood: | -9.6235 |
| No. Observations: | 42 | AIC: | 45.25 |
| Df Residuals: | 29 | BIC: | 67.84 |
| Df Model: | 12 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | -4.7031 | 2.279 | -2.063 | 0.048 | -9.365 | -0.041 |
| Length | 0.0267 | 0.034 | 0.774 | 0.445 | -0.044 | 0.097 |
| Headcirc | 0.0934 | 0.032 | 2.891 | 0.007 | 0.027 | 0.160 |
| Gestation | 0.0827 | 0.034 | 2.409 | 0.023 | 0.012 | 0.153 |
| smoker | -0.1953 | 0.185 | -1.058 | 0.299 | -0.573 | 0.182 |
| mage | -0.0226 | 0.020 | -1.115 | 0.274 | -0.064 | 0.019 |
| mnocig | 0.0007 | 0.008 | 0.099 | 0.922 | -0.015 | 0.016 |
| mheight | 1.471e+12 | nan | nan | nan | nan | nan |
| mppwt | 0.0084 | 0.012 | 0.680 | 0.502 | -0.017 | 0.034 |
| fage | 0.0132 | 0.018 | 0.736 | 0.468 | -0.023 | 0.050 |
| fedyrs | 0.0131 | 0.034 | 0.385 | 0.703 | -0.056 | 0.082 |
| fnocig | 0.0013 | 0.004 | 0.300 | 0.766 | -0.008 | 0.010 |
| fheight | 1.471e+12 | nan | nan | nan | nan | nan |
| aveheight | -2.943e+12 | nan | nan | nan | nan | nan |
| Omnibus: | 3.458 | Durbin-Watson: | 1.325 |
|---|---|---|---|
| Prob(Omnibus): | 0.177 | Jarque-Bera (JB): | 3.020 |
| Skew: | 0.565 | Prob(JB): | 0.221 |
| Kurtosis: | 2.329 | Cond. No. | 1.40e+16 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 2.16e-26. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.
As you see the $\beta_j$ that corresponds to the new variable cannot be calculated (hence the nan), since aveheight is linearly dependent on both mheight and fheight. At the same time, although the matrix $\mathbf{X}$ is not of full rank, statsmodel can produces solutions for the rest of the $\beta$'s using pseudo-inverse (with .fit(method='pinv')), and the fitted model is valid for prediction purposes. The predictions derived from this model will align with those obtained from fitting a linear regression model excluding the columns with nan coefficient.
This issue can be easily fixed, by removing one or more variables. Which variable will be removed depends on the context. For example, we need to (conceptually) answer the question on whether we prefer to have the aveheight or each parent’s height individually.
A harder issue to resolve is when two predictors are not exaclty linearly dependent, but approximately - this means that they are highly correlated, but not an exact function of one another. This is more challenging to deal with and we will discuss it later.
Test for the Significance of Predictors¶
We use the partial $F$ test $$F=\frac{\bigl(RSS_0 - RSS_{\alpha}\bigr)/(p_{\alpha} - p_0)}{RSS_{\alpha}/ \bigl(n-p_{\alpha} \bigr)}\, \sim \, F_{p_{\alpha} - p_0, n-p_{\alpha}}$$ to compare the full model with all available predictors versus the model that exclude father-related variables. That is, the hypothesis test is formulated as $$\begin{cases} &H_0: Y \sim 1+ \beta_{1} X_{1} + \ldots + \beta_{8}X_{8} \\ & H_{\alpha}: Y \sim 1+ \beta_1 X_1 + \ldots + \beta_{8}X_{8}+ \ldots + \beta_{12} X_{12} \end{cases} $$
# Full Model was fitted before and is stored in birthweight_mlr_full
# Reduced Model (without father-related variables)
reduced_formula1 = 'Birthweight ~ Length + Headcirc + Gestation + smoker + mage + mnocig + mheight + mppwt'
birthweight_mlr_reduced1 = smf.ols(reduced_formula1, data=birthweight).fit()
We can use one function to perform the test:
anova_results = sm.stats.anova_lm(
birthweight_mlr_reduced1, birthweight_mlr_full)
anova_results
| df_resid | ssr | df_diff | ss_diff | F | Pr(>F) | |
|---|---|---|---|---|---|---|
| 0 | 33.0 | 3.560996 | 0.0 | NaN | NaN | NaN |
| 1 | 29.0 | 3.238285 | 4.0 | 0.322711 | 0.722498 | 0.583652 |
The $F$ statistic value is 0.7225, and the corresponding $p$-value is 0.5837 which means that we fail to reject the null and conclude that the model under the $H_0$ is preferred. In other words, we prefer the smaller model which implies that the father-related predictors can be removed from the model.
We can also build the $F$ statistic, by extracting all necessary components:
RSS0 = birthweight_mlr_reduced1.ssr
RSSalpha = birthweight_mlr_full.ssr
print(f"RSS0 (Reduced): {RSS0:.4f}")
print(f"RSSalpha (Full): {RSSalpha:.4f}")
p0 = len(birthweight_mlr_reduced1.params)
palpha = len(birthweight_mlr_full.params)
n = birthweight.shape[0]
# Partial F test
F_manual = ((RSS0 - RSSalpha) / (palpha - p0)) / (RSSalpha / (n - palpha))
print(f"Manual F-statistic: {F_manual:.4f}")
# Critical value to compare with the F statistic.
alpha = 0.05
dfn = palpha - p0
dfd = n - palpha
critical_value = sp.stats.f.ppf(1 - alpha, dfn=dfn, dfd=dfd)
print(f"Critical F-value at alpha={alpha}: {critical_value:.4f}")
RSS0 (Reduced): 3.5610 RSSalpha (Full): 3.2383 Manual F-statistic: 0.7225 Critical F-value at alpha=0.05: 2.7014
the conclusion is the same since $F < F_{crit}$.
Testing for a Single $\beta_j$¶
Let's test for the significance of variable Length ($X_1$), i.e.
$$\begin{cases}
&H_0: Y \sim 1 + \qquad \quad \beta_2 X_2 + \ldots + \beta_{12}X_{12} \\
& H_{\alpha}: Y \sim 1+ \beta_1 X_1+ \beta_2 X_2 + \ldots + \beta_{12} X_{12} \\
\end{cases}
$$
We fit the full model and then compare by means of the partial $F$ test:
# Full Model was fitted before and is stored in birthweight_mlr_full
# Reduced Model (without 'Length')
reduced_formula2 = 'Birthweight ~ Headcirc + Gestation + smoker + mage + mnocig + mheight + mppwt + fage + fedyrs + fnocig + fheight'
birthweight_mlr_reduced2 = smf.ols(reduced_formula2, data=birthweight).fit()
# Partial F test
anova_single_var = sm.stats.anova_lm(
birthweight_mlr_reduced2, birthweight_mlr_full)
anova_single_var
| df_resid | ssr | df_diff | ss_diff | F | Pr(>F) | |
|---|---|---|---|---|---|---|
| 0 | 30.0 | 3.369906 | 0.0 | NaN | NaN | NaN |
| 1 | 29.0 | 3.238285 | 1.0 | 0.131622 | 1.178721 | 0.286559 |
# Compare the $F$ value above with the square of the $t$ test
# for the length variable in the output below:
birthweight_mlr_full.summary()
| Dep. Variable: | Birthweight | R-squared: | 0.783 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.694 |
| Method: | Least Squares | F-statistic: | 8.742 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 1.01e-06 |
| Time: | 01:31:58 | Log-Likelihood: | -5.7803 |
| No. Observations: | 42 | AIC: | 37.56 |
| Df Residuals: | 29 | BIC: | 60.15 |
| Df Model: | 12 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | -3.8123 | 2.080 | -1.833 | 0.077 | -8.067 | 0.442 |
| Length | 0.0342 | 0.031 | 1.086 | 0.287 | -0.030 | 0.098 |
| Headcirc | 0.0866 | 0.029 | 2.937 | 0.006 | 0.026 | 0.147 |
| Gestation | 0.0966 | 0.031 | 3.083 | 0.004 | 0.033 | 0.161 |
| smoker | -0.2439 | 0.169 | -1.447 | 0.159 | -0.589 | 0.101 |
| mage | -0.0190 | 0.019 | -1.025 | 0.314 | -0.057 | 0.019 |
| mnocig | 0.0011 | 0.007 | 0.155 | 0.878 | -0.013 | 0.015 |
| mheight | 0.0042 | 0.014 | 0.296 | 0.769 | -0.025 | 0.033 |
| mppwt | 0.0103 | 0.011 | 0.921 | 0.364 | -0.013 | 0.033 |
| fage | 0.0072 | 0.016 | 0.440 | 0.663 | -0.026 | 0.041 |
| fedyrs | 0.0032 | 0.031 | 0.104 | 0.918 | -0.060 | 0.066 |
| fnocig | 0.0034 | 0.004 | 0.858 | 0.398 | -0.005 | 0.012 |
| fheight | -0.0133 | 0.010 | -1.383 | 0.177 | -0.033 | 0.006 |
| Omnibus: | 5.590 | Durbin-Watson: | 1.680 |
|---|---|---|---|
| Prob(Omnibus): | 0.061 | Jarque-Bera (JB): | 3.108 |
| Skew: | 0.448 | Prob(JB): | 0.211 |
| Kurtosis: | 2.013 | Cond. No. | 1.07e+04 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.07e+04. This might indicate that there are
strong multicollinearity or other numerical problems.
# The square of the t-value is obtained by:
t_value_length = birthweight_mlr_full.tvalues['Length']
f_from_t_sq = t_value_length ** 2
f_from_t_sq
np.float64(1.1787209295796561)
Testing for all the predictors¶
The $F$ test statistic and corresponding $p$-value in the summary output correspond to the following hypothesis:
$$\begin{cases}
&H_0: Y \sim 1 \\
& H_{\alpha}: Y \sim 1+ \beta_1 X_1 + \ldots + \beta_p X_p \\
\end{cases}
$$
which is the so-called overall $F$ test.
print(f"Model degrees of freedom: {birthweight_mlr_full.df_model}")
print(f"Residual degrees of freedom.: {birthweight_mlr_full.df_resid}")
print(f"Overall F-statistic: {birthweight_mlr_full.fvalue:.4f}")
print(f"Overall F p-value: {birthweight_mlr_full.f_pvalue:.4e}")
Model degrees of freedom: 12.0 Residual degrees of freedom.: 29.0 Overall F-statistic: 8.7419 Overall F p-value: 1.0111e-06
In this case, the $F$ value is equal to 8.74 and it follows an $F_{12, 29}$ distribution.The corresponding $p$-value is equal to 1.0111e-06 which is much less than $\alpha=0.05$. This implies that we reject the null and conclude that the model with all the predictors is more adequate compared to the intercept-only model.
Categorical Variables¶
In the birthweight data set we only have binary variables, so for illustration purposes, we will create a new categorical variable, mage.group with 3 levels: $a$=below 20, $b$=20-30, and $c$=above 30:
birthweight_new = birthweight.copy()
conditions = [
(birthweight_new['mage'] < 20),
(birthweight_new['mage'] > 30)
]
choices = ['a', 'c']
birthweight_new['mage_group'] = np.select(conditions, choices, default='b')
# Let's fit the model with the age group variable and look at the design matrix:
formula_cat = 'Birthweight ~ ' + \
' + '.join(birthweight_new.columns.drop(['Birthweight', 'mage']))
birthweight_mlr_categorical = smf.ols(formula_cat, data=birthweight_new).fit()
# The design matrix shows the dummy variables created for 'mage_group'
design_matrix_cat = pd.DataFrame(
birthweight_mlr_categorical.model.exog,
columns=birthweight_mlr_categorical.model.exog_names
)
design_matrix_cat
| Intercept | mage_group[T.b] | mage_group[T.c] | Length | Headcirc | Gestation | smoker | mnocig | mheight | mppwt | fage | fedyrs | fnocig | fheight | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1.0 | 1.0 | 0.0 | 56.0 | 34.0 | 44.0 | 0.0 | 0.0 | 162.0 | 57.0 | 23.0 | 10.0 | 35.0 | 179.0 |
| 1 | 1.0 | 0.0 | 0.0 | 53.0 | 36.0 | 40.0 | 0.0 | 0.0 | 171.0 | 62.0 | 19.0 | 12.0 | 0.0 | 183.0 |
| 2 | 1.0 | 0.0 | 1.0 | 58.0 | 39.0 | 41.0 | 0.0 | 0.0 | 172.0 | 58.0 | 31.0 | 16.0 | 25.0 | 185.0 |
| 3 | 1.0 | 1.0 | 0.0 | 53.0 | 38.0 | 44.0 | 0.0 | 0.0 | 174.0 | 68.0 | 26.0 | 14.0 | 25.0 | 189.0 |
| 4 | 1.0 | 1.0 | 0.0 | 54.0 | 37.0 | 42.0 | 0.0 | 0.0 | 175.0 | 66.0 | 30.0 | 12.0 | 0.0 | 184.0 |
| 5 | 1.0 | 1.0 | 0.0 | 51.0 | 38.0 | 38.0 | 0.0 | 0.0 | 165.0 | 61.0 | 31.0 | 16.0 | 0.0 | 180.0 |
| 6 | 1.0 | 1.0 | 0.0 | 52.0 | 34.0 | 40.0 | 0.0 | 0.0 | 157.0 | 50.0 | 31.0 | 16.0 | 0.0 | 173.0 |
| 7 | 1.0 | 1.0 | 0.0 | 53.0 | 33.0 | 42.0 | 0.0 | 0.0 | 165.0 | 61.0 | 21.0 | 10.0 | 25.0 | 185.0 |
| 8 | 1.0 | 0.0 | 0.0 | 54.0 | 38.0 | 38.0 | 0.0 | 0.0 | 172.0 | 50.0 | 20.0 | 12.0 | 7.0 | 172.0 |
| 9 | 1.0 | 1.0 | 0.0 | 50.0 | 35.0 | 38.0 | 0.0 | 0.0 | 157.0 | 48.0 | 22.0 | 14.0 | 0.0 | 179.0 |
| 10 | 1.0 | 1.0 | 0.0 | 53.0 | 33.0 | 41.0 | 0.0 | 0.0 | 164.0 | 62.0 | 37.0 | 14.0 | 0.0 | 170.0 |
| 11 | 1.0 | 1.0 | 0.0 | 51.0 | 36.0 | 40.0 | 0.0 | 0.0 | 168.0 | 53.0 | 29.0 | 16.0 | 0.0 | 181.0 |
| 12 | 1.0 | 0.0 | 1.0 | 52.0 | 36.0 | 38.0 | 0.0 | 0.0 | 164.0 | 57.0 | 35.0 | 16.0 | 0.0 | 183.0 |
| 13 | 1.0 | 1.0 | 0.0 | 53.0 | 33.0 | 41.0 | 0.0 | 0.0 | 155.0 | 55.0 | 25.0 | 14.0 | 25.0 | 183.0 |
| 14 | 1.0 | 1.0 | 0.0 | 53.0 | 34.0 | 40.0 | 0.0 | 0.0 | 167.0 | 60.0 | 30.0 | 16.0 | 25.0 | 182.0 |
| 15 | 1.0 | 1.0 | 0.0 | 48.0 | 33.0 | 37.0 | 0.0 | 0.0 | 158.0 | 54.0 | 39.0 | 10.0 | 0.0 | 171.0 |
| 16 | 1.0 | 1.0 | 0.0 | 48.0 | 35.0 | 39.0 | 0.0 | 0.0 | 162.0 | 62.0 | 27.0 | 14.0 | 0.0 | 178.0 |
| 17 | 1.0 | 1.0 | 0.0 | 48.0 | 33.0 | 34.0 | 0.0 | 0.0 | 167.0 | 64.0 | 25.0 | 12.0 | 25.0 | 175.0 |
| 18 | 1.0 | 0.0 | 0.0 | 53.0 | 34.0 | 39.0 | 0.0 | 0.0 | 165.0 | 57.0 | 23.0 | 14.0 | 2.0 | 193.0 |
| 19 | 1.0 | 1.0 | 0.0 | 43.0 | 32.0 | 33.0 | 0.0 | 0.0 | 149.0 | 45.0 | 26.0 | 16.0 | 0.0 | 169.0 |
| 20 | 1.0 | 1.0 | 0.0 | 53.0 | 38.0 | 40.0 | 1.0 | 2.0 | 170.0 | 59.0 | 24.0 | 12.0 | 12.0 | 185.0 |
| 21 | 1.0 | 1.0 | 0.0 | 51.0 | 33.0 | 41.0 | 1.0 | 7.0 | 160.0 | 53.0 | 24.0 | 16.0 | 12.0 | 176.0 |
| 22 | 1.0 | 0.0 | 0.0 | 50.0 | 30.0 | 37.0 | 1.0 | 7.0 | 165.0 | 60.0 | 20.0 | 14.0 | 0.0 | 183.0 |
| 23 | 1.0 | 1.0 | 0.0 | 50.0 | 35.0 | 39.0 | 1.0 | 7.0 | 159.0 | 52.0 | 23.0 | 14.0 | 25.0 | 200.0 |
| 24 | 1.0 | 1.0 | 0.0 | 48.0 | 30.0 | 37.0 | 1.0 | 7.0 | 163.0 | 47.0 | 20.0 | 10.0 | 35.0 | 185.0 |
| 25 | 1.0 | 0.0 | 1.0 | 46.0 | 32.0 | 35.0 | 1.0 | 7.0 | 166.0 | 57.0 | 37.0 | 14.0 | 25.0 | 173.0 |
| 26 | 1.0 | 1.0 | 0.0 | 48.0 | 30.0 | 33.0 | 1.0 | 7.0 | 161.0 | 50.0 | 20.0 | 10.0 | 35.0 | 180.0 |
| 27 | 1.0 | 0.0 | 1.0 | 58.0 | 39.0 | 41.0 | 1.0 | 12.0 | 173.0 | 70.0 | 38.0 | 14.0 | 25.0 | 180.0 |
| 28 | 1.0 | 0.0 | 1.0 | 53.0 | 34.0 | 40.0 | 1.0 | 12.0 | 163.0 | 49.0 | 41.0 | 12.0 | 50.0 | 191.0 |
| 29 | 1.0 | 1.0 | 0.0 | 49.0 | 36.0 | 40.0 | 1.0 | 12.0 | 152.0 | 48.0 | 37.0 | 12.0 | 25.0 | 170.0 |
| 30 | 1.0 | 1.0 | 0.0 | 52.0 | 35.0 | 38.0 | 1.0 | 12.0 | 165.0 | 64.0 | 38.0 | 14.0 | 50.0 | 180.0 |
| 31 | 1.0 | 1.0 | 0.0 | 51.0 | 38.0 | 39.0 | 1.0 | 17.0 | 157.0 | 48.0 | 32.0 | 12.0 | 25.0 | 169.0 |
| 32 | 1.0 | 1.0 | 0.0 | 50.0 | 33.0 | 39.0 | 1.0 | 17.0 | 156.0 | 53.0 | 24.0 | 12.0 | 7.0 | 179.0 |
| 33 | 1.0 | 1.0 | 0.0 | 50.0 | 33.0 | 45.0 | 1.0 | 25.0 | 163.0 | 54.0 | 30.0 | 16.0 | 0.0 | 183.0 |
| 34 | 1.0 | 1.0 | 0.0 | 52.0 | 36.0 | 39.0 | 1.0 | 25.0 | 170.0 | 78.0 | 40.0 | 16.0 | 50.0 | 178.0 |
| 35 | 1.0 | 0.0 | 1.0 | 53.0 | 37.0 | 41.0 | 1.0 | 25.0 | 161.0 | 66.0 | 46.0 | 16.0 | 0.0 | 175.0 |
| 36 | 1.0 | 1.0 | 0.0 | 52.0 | 37.0 | 40.0 | 1.0 | 25.0 | 170.0 | 62.0 | 30.0 | 10.0 | 25.0 | 181.0 |
| 37 | 1.0 | 1.0 | 0.0 | 52.0 | 33.0 | 39.0 | 1.0 | 25.0 | 181.0 | 69.0 | 23.0 | 16.0 | 2.0 | 181.0 |
| 38 | 1.0 | 0.0 | 1.0 | 49.0 | 34.0 | 38.0 | 1.0 | 25.0 | 162.0 | 57.0 | 32.0 | 16.0 | 50.0 | 194.0 |
| 39 | 1.0 | 1.0 | 0.0 | 53.0 | 34.0 | 41.0 | 1.0 | 35.0 | 163.0 | 51.0 | 31.0 | 16.0 | 25.0 | 185.0 |
| 40 | 1.0 | 1.0 | 0.0 | 47.0 | 33.0 | 35.0 | 1.0 | 35.0 | 170.0 | 57.0 | 23.0 | 12.0 | 50.0 | 186.0 |
| 41 | 1.0 | 0.0 | 1.0 | 53.0 | 32.0 | 40.0 | 1.0 | 50.0 | 168.0 | 61.0 | 31.0 | 16.0 | 0.0 | 173.0 |
When fitting the regression model, statsmodels automatically converted the factor to the appropriate number of indicator/dummy variables which were included in the design matrix. We observe that level $a$ is the baseline (it doesn't have its own column) and the two generated columns are for levels $b$ and $c$.
In general, the reference level can be changed as well as the coding. Those changes will affect the estimated $\beta$ coefficients, but they will not affect the fitted or predicted values.
Mean Response Estimation & Prediction¶
In practice, we have target values for which we need to estimate the mean response of values for which we want to do prediction. Since this is not the case here, in order to illustrate both methods, we use as inputs the mean value of every predictor.
# The response variable is 'Birthweight', so we drop it from the mean calculation
mean_values = birthweight.drop('Birthweight', axis=1).mean()
Now, we create the $X$ DataFrame to use as an input in the predict() function:
x_new = pd.DataFrame(mean_values).T
Using the predict function we have:
# Use the predict function
prediction = birthweight_mlr_full.predict(x_new)
prediction
0 3.312857 dtype: float64
Training vs. Testing Error¶
In this example, and in linear regression models, as we introduce more variables, training error (measured by $\text{RSS}$ or $\text{MSE}$) consistently decreases. However, this reduction in training error does not guarantee a corresponding decrease in test error, which measures prediction accuracy on independent test data. To demonstrate this, we randomly split our dataset into $60\%$ training and $40\%$ test portions, progressively adding predictors.
# Using sklearn's train_test_split is the standard Python approach
X = birthweight.drop('Birthweight', axis=1)
y = birthweight['Birthweight']
p = X.shape[1]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)
train_data = pd.concat([X_train, y_train], axis=1)
test_data = pd.concat([X_test, y_test], axis=1)
train_mse = []
test_mse = []
predictor_cols = X.columns.tolist()
for i in range(p):
current_predictors = predictor_cols[:i + 1]
formula = 'Birthweight ~ ' + ' + '.join(current_predictors)
# Fit model on training data
model = smf.ols(formula, data=train_data).fit()
# Predict and calculate MSE for training set
train_pred = model.predict(X_train)
train_mse.append(np.mean((y_train - train_pred) ** 2))
# Predict and calculate MSE for test set
test_pred = model.predict(X_test)
test_mse.append(np.mean((y_test - test_pred) ** 2))
# Plotting Train vs. Test MSE
plt.figure(figsize=(6, 6))
sns.lineplot(x=range(1, len(predictor_cols) + 1), y=train_mse,
linestyle='--', marker='s', color='blue', label='Train MSE')
sns.lineplot(x=range(1, len(predictor_cols) + 1), y=test_mse,
linestyle='-', marker='o' \
'', color='red', label='Test MSE')
plt.xlabel("# of variables")
plt.ylabel("MSE")
plt.title("Train vs. Test MSE as Predictors are Added")
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)
plt.show()
In general, we expect that in each iteration, the blue line (representing the training error) consistently exhibits a monotonically decreasing trend, signifying the reduction of training error as more predictors are incorporated. On the other hand, the red line’s (testing error) trajectory may not uniformly decrease.
Test for Outliers¶
To identify outliers, we use the Bonferroni test. For that purpose, we need to compute the studentized residuals $t_{i}$ ( an externally standardized version of residuals ). Under the null hypothesis $H_{0}$, $t_{i} \sim t_{n-p-1}$, where $n$ is the sample size and $p$ is the number of predictors. We perform this test for all $n$ observations testing case at level $\alpha/n$:
# We work with the full model again.
n = birthweight.shape[0]
p = birthweight.shape[1]
We first compute the studentized residuals using the get_influence() function and the Bonferroni critical value using Student’s distribution:
# Computing Studentized Residuals
influence = birthweight_mlr_full.get_influence()
studentized_residuals = influence.resid_studentized_internal
# Critical value WITH Bonferroni correction
alpha = 0.05
bonferroni_cv = sp.stats.t.ppf(alpha / (2 * n), df=n - p - 1)
print(f"Bonferroni critical value: {bonferroni_cv:.4f}")
Bonferroni critical value: -3.6076
Now, we need to find which (if any) studentized residuals exceed the Bonferroni critical value:
# Sorting the residuals in descending order to find outliers (if any)
abs_studentized_resid = np.abs(studentized_residuals)
sorted_resid = np.sort(abs_studentized_resid)[::-1]
print("\nTop 10 absolute studentized residuals:")
print(sorted_resid[:10])
# Find outliers
outliers = sorted_resid[sorted_resid > np.abs(bonferroni_cv)]
print(f"\nNumber of outliers (residuals > critical value): {len(outliers)}")
Top 10 absolute studentized residuals: [1.96073359 1.81163213 1.76071978 1.63166956 1.59918995 1.57035592 1.4101976 1.38820038 1.32936059 1.30725325] Number of outliers (residuals > critical value): 0
Above, we computed a critical t-value of $\lvert-3.6076\rvert$ at $\alpha=0.05$. If an observation's studentized residual is higher (in absolute value) than this critical value, then this observation will be considered an outlier. According to this criterion, we can see that we don’t have any outliers in the data set, since none of the studentized residuals is higher than $3.6076$.
Checking Model Assumptions¶
We start by using residual plots and specifically the residuals against fitted values:
import seaborn as sns
fig, ax = plt.subplots(1, figsize=(6, 6))
fitted = birthweight_mlr_full.fittedvalues
resid = birthweight_mlr_full.resid
sns.residplot(
x=fitted,
y=resid,
lowess=True,
scatter_kws={'alpha': 0.5},
line_kws={'color': 'red', 'lw': 1, 'alpha': 0.8},
ax=ax)
# annotations
residual_abs = np.abs(resid)
abs_resid = np.flip(np.argsort(residual_abs), 0)
abs_resid_top_3 = abs_resid[:3]
for i in abs_resid_top_3:
ax.annotate(
i,
xy=(fitted[i], resid[i]),
color='C3')
ax.set_title('Residuals vs Fitted', fontweight="bold")
ax.set_xlabel('Fitted values')
ax.set_ylabel('Residuals')
Text(0, 0.5, 'Residuals')
The points on the plot are randomly scattered around the zero line, so we conclude that the constant variance assumption is satisfied.
We continue by checking the normality assumption:
fig, ax = plt.subplots(1, 2, figsize=(12, 6))
residual_norm = influence.resid_studentized_internal
QQ = ProbPlot(residual_norm)
fig = QQ.qqplot(line='45', alpha=0.5, lw=1, ax=ax[0])
sns.histplot(resid, stat='density', ax=ax[1]);
ax[0].set_title('Normal Q-Q', fontweight="bold")
ax[0].set_xlabel('Theoretical Quantiles')
ax[0].set_ylabel('Standardized Residuals');
ax[1].set_title('Histogram of Residuals', fontweight="bold")
Text(0.5, 1.0, 'Histogram of Residuals')
We seem to have some departures from the normality assumption in this case.
Collinearity: Car Seat Position Data (seatpos)¶
Car designers would find it helpful to know how different drivers will position the seat depending on their size and age. Researchers at the HuMoSim laboratory at the University of Michigan collected the following data on 38 drivers:
Age: Drivers age in yearsWeight: Drivers weight in lbsHtShoes: height with shoes in cmHt: height without shoes in cmSeated: seated height in cmArm: lower arm length in cmThigh: thigh length in cmLeg: lower leg length in cmhipcenter: horizontal distance of the midpoint of the hips from a fixed location in the car in mm
This dataset originates from the faraway R package and a .csv version can be downloaded from Julian Faraway's Linear Models with Python repository.
seatpos = pd.read_csv("seatpos.csv")
# Fit the FULL model
var_names = [x for x in seatpos.columns if x != 'hipcenter']
fml_full = "hipcenter" + "~" + "+".join(x for x in var_names)
position_full = smf.ols(fml_full, data=seatpos).fit()
position_full.summary()
| Dep. Variable: | hipcenter | R-squared: | 0.687 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.600 |
| Method: | Least Squares | F-statistic: | 7.940 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 1.31e-05 |
| Time: | 01:31:59 | Log-Likelihood: | -186.73 |
| No. Observations: | 38 | AIC: | 391.5 |
| Df Residuals: | 29 | BIC: | 406.2 |
| Df Model: | 8 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | 436.4321 | 166.572 | 2.620 | 0.014 | 95.755 | 777.109 |
| Age | 0.7757 | 0.570 | 1.360 | 0.184 | -0.391 | 1.942 |
| Weight | 0.0263 | 0.331 | 0.080 | 0.937 | -0.651 | 0.703 |
| HtShoes | -2.6924 | 9.753 | -0.276 | 0.784 | -22.640 | 17.255 |
| Ht | 0.6013 | 10.130 | 0.059 | 0.953 | -20.117 | 21.319 |
| Seated | 0.5338 | 3.762 | 0.142 | 0.888 | -7.160 | 8.228 |
| Arm | -1.3281 | 3.900 | -0.341 | 0.736 | -9.305 | 6.649 |
| Thigh | -1.1431 | 2.660 | -0.430 | 0.671 | -6.583 | 4.297 |
| Leg | -6.4390 | 4.714 | -1.366 | 0.182 | -16.080 | 3.202 |
| Omnibus: | 0.543 | Durbin-Watson: | 1.769 |
|---|---|---|---|
| Prob(Omnibus): | 0.762 | Jarque-Bera (JB): | 0.664 |
| Skew: | 0.157 | Prob(JB): | 0.717 |
| Kurtosis: | 2.434 | Cond. No. | 8.44e+03 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 8.44e+03. This might indicate that there are
strong multicollinearity or other numerical problems.
Note that the $p$-value of the overall $F$ test is very small (1.31e-05), which means that at least one of the predictors is statistically significant, but at the same time none of the predictors is statistically significant according to the individual $t$ tests. This is strange and definitely a red flag that we may have collinearity issues.
To investigate the presence of high correlated predictors as well as the linear correlation of the response with each of the predictors, we start by checking the correlation matrix:
seatpos.corr().round(2)
| Age | Weight | HtShoes | Ht | Seated | Arm | Thigh | Leg | hipcenter | |
|---|---|---|---|---|---|---|---|---|---|
| Age | 1.00 | 0.08 | -0.08 | -0.09 | -0.17 | 0.36 | 0.09 | -0.04 | 0.21 |
| Weight | 0.08 | 1.00 | 0.83 | 0.83 | 0.78 | 0.70 | 0.57 | 0.78 | -0.64 |
| HtShoes | -0.08 | 0.83 | 1.00 | 1.00 | 0.93 | 0.75 | 0.72 | 0.91 | -0.80 |
| Ht | -0.09 | 0.83 | 1.00 | 1.00 | 0.93 | 0.75 | 0.73 | 0.91 | -0.80 |
| Seated | -0.17 | 0.78 | 0.93 | 0.93 | 1.00 | 0.63 | 0.61 | 0.81 | -0.73 |
| Arm | 0.36 | 0.70 | 0.75 | 0.75 | 0.63 | 1.00 | 0.67 | 0.75 | -0.59 |
| Thigh | 0.09 | 0.57 | 0.72 | 0.73 | 0.61 | 0.67 | 1.00 | 0.65 | -0.59 |
| Leg | -0.04 | 0.78 | 0.91 | 0.91 | 0.81 | 0.75 | 0.65 | 1.00 | -0.79 |
| hipcenter | 0.21 | -0.64 | -0.80 | -0.80 | -0.73 | -0.59 | -0.59 | -0.79 | 1.00 |
We observe that the response is highly correlated with most of the variables which justifies the low $p$-value in the overall $F$ test. However, we have highly correlated predictors (e.g. Corr(Leg, HtShoes)=0.91) which means that we will have collinearity.
Let's look at the condition number of the $\mathbf{X}^T\mathbf{X}$ matrix. Recall that the condition number is not scale invariant. This implies that calculating it before standardizing the matrix might lead to incorrect conclusions. So, let's standardize it first:
# Extract the design matrix and remove the column of 1s that corresponds to the intercept.
X = position_full.model.exog[:, 1:]
# Standardize the matrix
scaler = StandardScaler()
X = scaler.fit_transform(X)
# show the variance
X.var(axis=0, ddof=0)
array([1., 1., 1., 1., 1., 1., 1., 1.])
Compute the condition number using the standardized matrix:
# Compute the eigenvalues of the correlation matrix (X^T*X for standardized data)
eigenvalues, _ = np.linalg.eig(X.T @ X)
print("Eigenvalues of the correlation matrix:")
print(np.sort(eigenvalues)[::-1])
# Compute the condition number:
condition_number = np.sqrt(eigenvalues.max() / eigenvalues.min())
print(f"Condition Number: {condition_number:.4f}")
Eigenvalues of the correlation matrix: [2.15581055e+02 4.69978684e+01 1.76222508e+01 9.15641723e+00 7.38034342e+00 5.28861961e+00 1.91309203e+00 6.03530674e-02] Condition Number: 59.7662
The condition number is 59.77, larger than 30, so we conclude that collinearity is present.
Let's also check the Variance Inflation Factor (VIF):
variance_inflation_factor(X, 1)
np.float64(3.647030128812391)
p = X.shape[1]
vif_data = np.array([variance_inflation_factor(X, i) for i in range(p)])
display(vif_data)
np.sqrt(307.43)
array([ 1.99793148, 3.64703013, 307.42937802, 333.13783239,
8.9510538 , 4.49636844, 2.76288552, 6.69429122])
np.float64(17.5336818723279)
Note that the standard error for the coefficient associated with HtShoes is 17.53 times larger than it would have been without collinearity.
The next step is to investigate which variable to remove. To do so, we study the pairwise correlations and perform partial $F$-tests as follows:
position_red1 = smf.ols(
'hipcenter ~ Age + Weight + Ht + Seated', data=seatpos).fit()
position_red1.summary()
| Dep. Variable: | hipcenter | R-squared: | 0.660 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.619 |
| Method: | Least Squares | F-statistic: | 16.01 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 2.22e-07 |
| Time: | 01:31:59 | Log-Likelihood: | -188.28 |
| No. Observations: | 38 | AIC: | 386.6 |
| Df Residuals: | 33 | BIC: | 394.8 |
| Df Model: | 4 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | 478.6589 | 159.734 | 2.997 | 0.005 | 153.678 | 803.639 |
| Age | 0.5840 | 0.426 | 1.372 | 0.179 | -0.282 | 1.450 |
| Weight | -0.0154 | 0.316 | -0.049 | 0.962 | -0.659 | 0.628 |
| Ht | -4.9902 | 1.644 | -3.036 | 0.005 | -8.335 | -1.646 |
| Seated | 2.0463 | 3.413 | 0.600 | 0.553 | -4.897 | 8.990 |
| Omnibus: | 0.512 | Durbin-Watson: | 1.672 |
|---|---|---|---|
| Prob(Omnibus): | 0.774 | Jarque-Bera (JB): | 0.435 |
| Skew: | -0.244 | Prob(JB): | 0.805 |
| Kurtosis: | 2.810 | Cond. No. | 6.70e+03 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 6.7e+03. This might indicate that there are
strong multicollinearity or other numerical problems.
position_red2 = smf.ols('hipcenter ~ Ht', data=seatpos).fit()
display(position_red2.summary())
| Dep. Variable: | hipcenter | R-squared: | 0.638 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.628 |
| Method: | Least Squares | F-statistic: | 63.53 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 1.83e-09 |
| Time: | 01:31:59 | Log-Likelihood: | -189.45 |
| No. Observations: | 38 | AIC: | 382.9 |
| Df Residuals: | 36 | BIC: | 386.2 |
| Df Model: | 1 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | 556.2553 | 90.670 | 6.135 | 0.000 | 372.367 | 740.144 |
| Ht | -4.2650 | 0.535 | -7.970 | 0.000 | -5.350 | -3.180 |
| Omnibus: | 1.707 | Durbin-Watson: | 1.890 |
|---|---|---|---|
| Prob(Omnibus): | 0.426 | Jarque-Bera (JB): | 1.028 |
| Skew: | -0.395 | Prob(JB): | 0.598 |
| Kurtosis: | 3.154 | Cond. No. | 2.60e+03 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 2.6e+03. This might indicate that there are
strong multicollinearity or other numerical problems.
anova_seatpos = sm.stats.anova_lm(position_red2, position_red1)
anova_seatpos
| df_resid | ssr | df_diff | ss_diff | F | Pr(>F) | |
|---|---|---|---|---|---|---|
| 0 | 36.0 | 47615.793248 | 0.0 | NaN | NaN | NaN |
| 1 | 33.0 | 44774.245915 | 3.0 | 2841.547333 | 0.698103 | 0.559892 |
Based on the $F$ test provided in the ANOVA table, we conclude that the reduced model with Ht as the only variable is better than the model that includes Age, Weight, Ht and Seated.