Section 2.11 Python Code and Analysis¶

Birthweight Data Set Example¶

The birthweight.csv data set contains data from a study on the birth weight of 42 babies. The variables in the data set are the following:

  • ID: Unique Identification number of a baby
  • Length: Length of the baby at time of birth in cm ($X_1$)
  • Birthweight: Weight of the baby at time of birth in kg ($Y$)
  • Headcirc: Head circumference of the baby at time of birth in cm ($X_2$)
  • Gestation: Completed weeks of gestation ($X_3$)
  • smoker: Mother is/is not a smoker {0: No, 1: Yes} ($X_4$)
  • mage: Mother's age at time of birth ($X_5$)
  • mnocig: Mother's number of cigarettes smoked per day ($X_6$)
  • mheight: Mother's height in cm ($X_7$)
  • mppwt: Mother's pre-pregnancy weight ($X_8$)
  • fage: Father's age at time of birth ($X_9$)
  • fedyrs: Father's years of education ($X_{10}$)
  • fnocig: Father's number of cigarettes smoked per day ($X_{11}$)
  • fheight: Father's height ($X_{12}$)
In [43]:
## Import necessary libraries

import pandas as pd
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import seaborn as sns

import statsmodels.api as sm
import statsmodels.formula.api as smf
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.graphics.gofplots import ProbPlot

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

sns.set()
In [44]:
# load data
birthweight = pd.read_csv("Birthweight.csv")
print("Shape:", birthweight.shape)
birthweight.head()
Shape: (42, 14)
Out[44]:
ID Length Birthweight Headcirc Gestation smoker mage mnocig mheight mppwt fage fedyrs fnocig fheight
0 1360 56 4.55 34 44 0 20 0 162 57 23 10 35 179
1 1016 53 4.32 36 40 0 19 0 171 62 19 12 0 183
2 462 58 4.10 39 41 0 35 0 172 58 31 16 25 185
3 1187 53 4.07 38 44 0 20 0 174 68 26 14 25 189
4 553 54 3.94 37 42 0 24 0 175 66 30 12 0 184

The ID variable is not of interest, so we remove it from the data frame:

In [45]:
birthweight = birthweight.drop('ID', axis=1)

Fitting a Multiple Linear Regression¶

We begin by fitting a linear regression model that includes all predictors and examine the least-squares summary:

In [46]:
# statsmodels support R-style formula but not the period operator
# so we must construct the full formula manually
var_names = birthweight.columns.drop(['Birthweight'])
fml_full = 'Birthweight ~ ' + ' + '.join(x for x in var_names)
In [47]:
birthweight_mlr_full = smf.ols(formula=fml_full, data=birthweight).fit()
birthweight_mlr_full.summary()
Out[47]:
OLS Regression Results
Dep. Variable: Birthweight R-squared: 0.783
Model: OLS Adj. R-squared: 0.694
Method: Least Squares F-statistic: 8.742
Date: Fri, 21 Aug 2026 Prob (F-statistic): 1.01e-06
Time: 01:31:58 Log-Likelihood: -5.7803
No. Observations: 42 AIC: 37.56
Df Residuals: 29 BIC: 60.15
Df Model: 12
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept -3.8123 2.080 -1.833 0.077 -8.067 0.442
Length 0.0342 0.031 1.086 0.287 -0.030 0.098
Headcirc 0.0866 0.029 2.937 0.006 0.026 0.147
Gestation 0.0966 0.031 3.083 0.004 0.033 0.161
smoker -0.2439 0.169 -1.447 0.159 -0.589 0.101
mage -0.0190 0.019 -1.025 0.314 -0.057 0.019
mnocig 0.0011 0.007 0.155 0.878 -0.013 0.015
mheight 0.0042 0.014 0.296 0.769 -0.025 0.033
mppwt 0.0103 0.011 0.921 0.364 -0.013 0.033
fage 0.0072 0.016 0.440 0.663 -0.026 0.041
fedyrs 0.0032 0.031 0.104 0.918 -0.060 0.066
fnocig 0.0034 0.004 0.858 0.398 -0.005 0.012
fheight -0.0133 0.010 -1.383 0.177 -0.033 0.006
Omnibus: 5.590 Durbin-Watson: 1.680
Prob(Omnibus): 0.061 Jarque-Bera (JB): 3.108
Skew: 0.448 Prob(JB): 0.211
Kurtosis: 2.013 Cond. No. 1.07e+04


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.07e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

The objects returned by smf.ols can be inspected with:

In [48]:
[x for x in dir(birthweight_mlr_full) if not x.startswith('_')]
Out[48]:
['HC0_se',
 'HC1_se',
 'HC2_se',
 'HC3_se',
 'aic',
 'bic',
 'bse',
 'centered_tss',
 'compare_f_test',
 'compare_lm_test',
 'compare_lr_test',
 'condition_number',
 'conf_int',
 'conf_int_el',
 'cov_HC0',
 'cov_HC1',
 'cov_HC2',
 'cov_HC3',
 'cov_kwds',
 'cov_params',
 'cov_type',
 'df_model',
 'df_resid',
 'diagn',
 'eigenvals',
 'el_test',
 'ess',
 'f_pvalue',
 'f_test',
 'fittedvalues',
 'fvalue',
 'get_influence',
 'get_prediction',
 'get_robustcov_results',
 'info_criteria',
 'initialize',
 'k_constant',
 'llf',
 'load',
 'model',
 'mse_model',
 'mse_resid',
 'mse_total',
 'nobs',
 'normalized_cov_params',
 'outlier_test',
 'params',
 'predict',
 'pvalues',
 'remove_data',
 'resid',
 'resid_pearson',
 'rsquared',
 'rsquared_adj',
 'save',
 'scale',
 'ssr',
 'summary',
 'summary2',
 't_test',
 't_test_pairwise',
 'tvalues',
 'uncentered_tss',
 'use_t',
 'wald_test',
 'wald_test_terms',
 'wresid']

The design matrix is obtained by

In [49]:
# The design matrix (including intercept) is stored in the .model.exog attribute
design_birthweight = birthweight_mlr_full.model.exog

The estimated regression coefficients (including the intercept) are

In [50]:
# The coefficients are stored in the .params attribute
birthweight_mlr_full.params
Out[50]:
Intercept   -3.812311
Length       0.034152
Headcirc     0.086594
Gestation    0.096565
smoker      -0.243895
mage        -0.019004
mnocig       0.001064
mheight      0.004187
mppwt        0.010346
fage         0.007187
fedyrs       0.003220
fnocig       0.003445
fheight     -0.013293
dtype: float64

Fitted values and residuals are extracted as

In [51]:
full_model_residuals = birthweight_mlr_full.resid
full_model_fitted = birthweight_mlr_full.fittedvalues

Residual standard error and R2¶

The residual variance estimator is

$$\hat{\sigma}^2 = \frac{1}{n-p-1}\sum_i r_i^2 = \frac{\text{RSS}}{n-p-1}$$

To obtain an estimator for $\hat{\sigma}$, we can either use the formula (from the notes):

In [52]:
n = birthweight.shape[0]
p = birthweight.shape[1] - 1  # Number of predictors (excluding response)
rse = np.sqrt(np.sum(full_model_residuals ** 2) / (n - p - 1))
print(f"Manual RSE calculation: {rse:.6f}")
Manual RSE calculation: 0.334163

or we can extract it from the ols object:

In [53]:
# The model's 'scale' attribute is the estimated error variance (sigma-squared)
sigma_from_model = np.sqrt(birthweight_mlr_full.scale)
print(f"RSE from model (sqrt of scale): {sigma_from_model:.6f}")
RSE from model (sqrt of scale): 0.334163

The coefficient of determination computes as: $$ \frac{\sum_i (\hat{y}_i - \bar{y})^2}{\sum_i (y_i - \bar{y})^2} = 1 - \frac{\text{RSS}}{\text{TSS}}$$ We can extract $R^2$ from the output or compute it:

In [54]:
print(f"R-squared from model: {birthweight_mlr_full.rsquared:.6f}")

rss = np.sum(full_model_residuals ** 2)
tss = np.sum((birthweight['Birthweight'] -
             birthweight['Birthweight'].mean()) ** 2)
r_squared_manual1 = 1 - (rss / tss)
r_squared_manual2 = 1 - np.var(full_model_residuals) / \
    np.var(birthweight['Birthweight'])

print(
    f"Manual R-squared calculations: {r_squared_manual1:.6f} and {r_squared_manual2:.6f}")
R-squared from model: 0.783425
Manual R-squared calculations: 0.783425 and 0.783425

Invariance of $R^2$ & Fitted Values¶

We verify the invariance property of 𝑅2 by centering one of the predictors:

In [55]:
# Center the Length variable
birthweight_new = birthweight.copy()
birthweight_new['Length_centered'] = birthweight_new['Length'] - \
    birthweight_new['Length'].mean()

# Model with original `Length` (same as birthweight_mlr_full)
model_orig = smf.ols(fml_full, data=birthweight).fit()

# Model with centered `Length`
# We create a formula that includes all columns from birthweight_new except 'Length'
predictors_centered = birthweight_new.columns.drop(['Birthweight', 'Length'])
fml_centered = 'Birthweight ~ ' + ' + '.join(predictors_centered)
model_centered = smf.ols(fml_centered, data=birthweight_new).fit()

print(f"R-squared with original `Length`: {model_orig.rsquared:.6f}")
print(f"R-squared with centered `Length`: {model_centered.rsquared:.6f}")
R-squared with original `Length`: 0.783425
R-squared with centered `Length`: 0.783425

The fitted values from the two models are essentially identical (any tiny difference is due to floating-point rounding):

In [56]:
fit_diff = np.sum(model_orig.fittedvalues - model_centered.fittedvalues)
print(f"Sum of differences in fitted values: {fit_diff}")
Sum of differences in fitted values: 6.932232565759477e-13

$R^2$ and $\text{RSS}$ behavior when adding more $X$'s¶

We can also visualize how the $R^2$ and $\text{RSS}$ change as a function of the number of variables in the model to observe that adding a new predictor, even if it is randomly generated and has nothing to do with $Y$ will decrease $\text{RSS}$ and therefore increase $R^2$.

So, start with the null model that contains only the intercept and then we add variables one at a time, keeping the $R^2$ and $\text{RSS}$ that we will plot in the end. We will also randomly generate a new variable randomvar to illustrate that even an unrelevant variable has an effect on both $R^2$ and $\text{RSS}$ :

In [57]:
birthweight_new = birthweight.copy()
birthweight_new['randomvar'] = np.random.normal(size=birthweight_new.shape[0])
predictors = birthweight_new.columns.drop(['Birthweight']).tolist()

rsq = []
rss = []

# Model with only intercept
intercept_model = smf.ols('Birthweight ~ 1', data=birthweight_new).fit()
rsq.append(intercept_model.rsquared)
rss.append(intercept_model.ssr)  # ssr is Sum of Squared Residuals

# Sequentially add predictors
for i in range(len(predictors)):
    current_predictors = predictors[:(i + 1)]
    formula = 'Birthweight ~ ' + ' + '.join(current_predictors)
    model = smf.ols(formula, data=birthweight_new).fit()
    rsq.append(model.rsquared)
    rss.append(model.ssr)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))

sns.lineplot(x=range(len(rsq)), y=rsq, lw=2, color='orange', marker='o', ax=ax1)
ax1.set_xlabel("No. of Predictors")
ax1.set_ylabel("R-square")
ax1.set_title("R-squared vs. Number of Predictors")

sns.lineplot(x=range(len(rss)), y=rss, lw=2, color='blue', marker='o', ax=ax2)
ax2.set_xlabel("No. of Predictors")
ax2.set_ylabel("RSS")
ax2.set_title("RSS vs. Number of Predictors")

plt.show()
No description has been provided for this image

This is to illustrate why $R^2$ and $\text{RSS}$ are not good metrics when it comes to model selection. They are useful, because they helps us understand the quality of the model or $\text{RSS}$ is a key ingredient in constructing hypothesis tests, but they are never used for model selection.


Rank Deficiency¶

We have rank deficiency when $\mathbf{X}$ is not of full rank which leads to the $(\mathbf{X}^\top\mathbf{X})$ matrix not being invertible.

Here we illustrate a scenario where it is easy to identify the cause of the problem. (The error message you see below the code is because of that.)

In [58]:
# Create a new variable: the average height of both parents
birthweight_new = birthweight.copy()
birthweight_new['aveheight'] = (
    birthweight_new['mheight'] + birthweight_new['fheight']) / 2
var_names = birthweight_new.columns.drop(['Birthweight'])
fml_full_new = "Birthweight" + "~" + "+".join(x for x in var_names)

# Regress birthweight against all predictors
model_rankdef = smf.ols(fml_full_new, data=birthweight_new).fit(method='qr')
model_rankdef.summary()
/Users/alexandrachronopoulou/psl-book-env/lib/python3.11/site-packages/statsmodels/regression/linear_model.py:1884: RuntimeWarning: invalid value encountered in sqrt
  return np.sqrt(np.diag(self.cov_params()))
Out[58]:
OLS Regression Results
Dep. Variable: Birthweight R-squared: 0.740
Model: OLS Adj. R-squared: 0.632
Method: Least Squares F-statistic: 6.876
Date: Fri, 21 Aug 2026 Prob (F-statistic): 1.11e-05
Time: 01:31:58 Log-Likelihood: -9.6235
No. Observations: 42 AIC: 45.25
Df Residuals: 29 BIC: 67.84
Df Model: 12
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept -4.7031 2.279 -2.063 0.048 -9.365 -0.041
Length 0.0267 0.034 0.774 0.445 -0.044 0.097
Headcirc 0.0934 0.032 2.891 0.007 0.027 0.160
Gestation 0.0827 0.034 2.409 0.023 0.012 0.153
smoker -0.1953 0.185 -1.058 0.299 -0.573 0.182
mage -0.0226 0.020 -1.115 0.274 -0.064 0.019
mnocig 0.0007 0.008 0.099 0.922 -0.015 0.016
mheight 1.471e+12 nan nan nan nan nan
mppwt 0.0084 0.012 0.680 0.502 -0.017 0.034
fage 0.0132 0.018 0.736 0.468 -0.023 0.050
fedyrs 0.0131 0.034 0.385 0.703 -0.056 0.082
fnocig 0.0013 0.004 0.300 0.766 -0.008 0.010
fheight 1.471e+12 nan nan nan nan nan
aveheight -2.943e+12 nan nan nan nan nan
Omnibus: 3.458 Durbin-Watson: 1.325
Prob(Omnibus): 0.177 Jarque-Bera (JB): 3.020
Skew: 0.565 Prob(JB): 0.221
Kurtosis: 2.329 Cond. No. 1.40e+16


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 2.16e-26. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.

As you see the $\beta_j$ that corresponds to the new variable cannot be calculated (hence the nan), since aveheight is linearly dependent on both mheight and fheight. At the same time, although the matrix $\mathbf{X}$ is not of full rank, statsmodel can produces solutions for the rest of the $\beta$'s using pseudo-inverse (with .fit(method='pinv')), and the fitted model is valid for prediction purposes. The predictions derived from this model will align with those obtained from fitting a linear regression model excluding the columns with nan coefficient.

This issue can be easily fixed, by removing one or more variables. Which variable will be removed depends on the context. For example, we need to (conceptually) answer the question on whether we prefer to have the aveheight or each parent’s height individually.

A harder issue to resolve is when two predictors are not exaclty linearly dependent, but approximately - this means that they are highly correlated, but not an exact function of one another. This is more challenging to deal with and we will discuss it later.


Test for the Significance of Predictors¶

We use the partial $F$ test $$F=\frac{\bigl(RSS_0 - RSS_{\alpha}\bigr)/(p_{\alpha} - p_0)}{RSS_{\alpha}/ \bigl(n-p_{\alpha} \bigr)}\, \sim \, F_{p_{\alpha} - p_0, n-p_{\alpha}}$$ to compare the full model with all available predictors versus the model that exclude father-related variables. That is, the hypothesis test is formulated as $$\begin{cases} &H_0: Y \sim 1+ \beta_{1} X_{1} + \ldots + \beta_{8}X_{8} \\ & H_{\alpha}: Y \sim 1+ \beta_1 X_1 + \ldots + \beta_{8}X_{8}+ \ldots + \beta_{12} X_{12} \end{cases} $$

In [59]:
# Full Model was fitted before and is stored in birthweight_mlr_full

# Reduced Model (without father-related variables)
reduced_formula1 = 'Birthweight ~ Length + Headcirc + Gestation + smoker + mage + mnocig + mheight + mppwt'
birthweight_mlr_reduced1 = smf.ols(reduced_formula1, data=birthweight).fit()

We can use one function to perform the test:

In [60]:
anova_results = sm.stats.anova_lm(
    birthweight_mlr_reduced1, birthweight_mlr_full)
anova_results
Out[60]:
df_resid ssr df_diff ss_diff F Pr(>F)
0 33.0 3.560996 0.0 NaN NaN NaN
1 29.0 3.238285 4.0 0.322711 0.722498 0.583652

The $F$ statistic value is 0.7225, and the corresponding $p$-value is 0.5837 which means that we fail to reject the null and conclude that the model under the $H_0$ is preferred. In other words, we prefer the smaller model which implies that the father-related predictors can be removed from the model.

We can also build the $F$ statistic, by extracting all necessary components:

In [61]:
RSS0 = birthweight_mlr_reduced1.ssr
RSSalpha = birthweight_mlr_full.ssr
print(f"RSS0 (Reduced): {RSS0:.4f}")
print(f"RSSalpha (Full): {RSSalpha:.4f}")

p0 = len(birthweight_mlr_reduced1.params)
palpha = len(birthweight_mlr_full.params)
n = birthweight.shape[0]

# Partial F test
F_manual = ((RSS0 - RSSalpha) / (palpha - p0)) / (RSSalpha / (n - palpha))
print(f"Manual F-statistic: {F_manual:.4f}")

# Critical value to compare with the F statistic.
alpha = 0.05
dfn = palpha - p0
dfd = n - palpha
critical_value = sp.stats.f.ppf(1 - alpha, dfn=dfn, dfd=dfd)
print(f"Critical F-value at alpha={alpha}: {critical_value:.4f}")
RSS0 (Reduced): 3.5610
RSSalpha (Full): 3.2383
Manual F-statistic: 0.7225
Critical F-value at alpha=0.05: 2.7014

the conclusion is the same since $F < F_{crit}$.

Testing for a Single $\beta_j$¶

Let's test for the significance of variable Length ($X_1$), i.e. $$\begin{cases} &H_0: Y \sim 1 + \qquad \quad \beta_2 X_2 + \ldots + \beta_{12}X_{12} \\ & H_{\alpha}: Y \sim 1+ \beta_1 X_1+ \beta_2 X_2 + \ldots + \beta_{12} X_{12} \\ \end{cases} $$ We fit the full model and then compare by means of the partial $F$ test:

In [62]:
# Full Model was fitted before and is stored in birthweight_mlr_full

# Reduced Model (without 'Length')
reduced_formula2 = 'Birthweight ~ Headcirc + Gestation + smoker + mage + mnocig + mheight + mppwt + fage + fedyrs + fnocig + fheight'
birthweight_mlr_reduced2 = smf.ols(reduced_formula2, data=birthweight).fit()

# Partial F test
anova_single_var = sm.stats.anova_lm(
    birthweight_mlr_reduced2, birthweight_mlr_full)
anova_single_var
Out[62]:
df_resid ssr df_diff ss_diff F Pr(>F)
0 30.0 3.369906 0.0 NaN NaN NaN
1 29.0 3.238285 1.0 0.131622 1.178721 0.286559
In [63]:
# Compare the $F$ value above with the square of the $t$ test 
# for the length variable in the output below:
birthweight_mlr_full.summary()
Out[63]:
OLS Regression Results
Dep. Variable: Birthweight R-squared: 0.783
Model: OLS Adj. R-squared: 0.694
Method: Least Squares F-statistic: 8.742
Date: Fri, 21 Aug 2026 Prob (F-statistic): 1.01e-06
Time: 01:31:58 Log-Likelihood: -5.7803
No. Observations: 42 AIC: 37.56
Df Residuals: 29 BIC: 60.15
Df Model: 12
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept -3.8123 2.080 -1.833 0.077 -8.067 0.442
Length 0.0342 0.031 1.086 0.287 -0.030 0.098
Headcirc 0.0866 0.029 2.937 0.006 0.026 0.147
Gestation 0.0966 0.031 3.083 0.004 0.033 0.161
smoker -0.2439 0.169 -1.447 0.159 -0.589 0.101
mage -0.0190 0.019 -1.025 0.314 -0.057 0.019
mnocig 0.0011 0.007 0.155 0.878 -0.013 0.015
mheight 0.0042 0.014 0.296 0.769 -0.025 0.033
mppwt 0.0103 0.011 0.921 0.364 -0.013 0.033
fage 0.0072 0.016 0.440 0.663 -0.026 0.041
fedyrs 0.0032 0.031 0.104 0.918 -0.060 0.066
fnocig 0.0034 0.004 0.858 0.398 -0.005 0.012
fheight -0.0133 0.010 -1.383 0.177 -0.033 0.006
Omnibus: 5.590 Durbin-Watson: 1.680
Prob(Omnibus): 0.061 Jarque-Bera (JB): 3.108
Skew: 0.448 Prob(JB): 0.211
Kurtosis: 2.013 Cond. No. 1.07e+04


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.07e+04. This might indicate that there are
strong multicollinearity or other numerical problems.
In [64]:
# The square of the t-value is obtained by:
t_value_length = birthweight_mlr_full.tvalues['Length']
f_from_t_sq = t_value_length ** 2
f_from_t_sq
Out[64]:
np.float64(1.1787209295796561)

Testing for all the predictors¶

The $F$ test statistic and corresponding $p$-value in the summary output correspond to the following hypothesis: $$\begin{cases} &H_0: Y \sim 1 \\ & H_{\alpha}: Y \sim 1+ \beta_1 X_1 + \ldots + \beta_p X_p \\ \end{cases} $$ which is the so-called overall $F$ test.

In [65]:
print(f"Model degrees of freedom: {birthweight_mlr_full.df_model}")
print(f"Residual degrees of freedom.: {birthweight_mlr_full.df_resid}")
print(f"Overall F-statistic: {birthweight_mlr_full.fvalue:.4f}")
print(f"Overall F p-value: {birthweight_mlr_full.f_pvalue:.4e}")
Model degrees of freedom: 12.0
Residual degrees of freedom.: 29.0
Overall F-statistic: 8.7419
Overall F p-value: 1.0111e-06

In this case, the $F$ value is equal to 8.74 and it follows an $F_{12, 29}$ distribution.The corresponding $p$-value is equal to 1.0111e-06 which is much less than $\alpha=0.05$. This implies that we reject the null and conclude that the model with all the predictors is more adequate compared to the intercept-only model.


Categorical Variables¶

In the birthweight data set we only have binary variables, so for illustration purposes, we will create a new categorical variable, mage.group with 3 levels: $a$=below 20, $b$=20-30, and $c$=above 30:

In [66]:
birthweight_new = birthweight.copy()

conditions = [
    (birthweight_new['mage'] < 20),
    (birthweight_new['mage'] > 30)
]
choices = ['a', 'c']
birthweight_new['mage_group'] = np.select(conditions, choices, default='b')

# Let's fit the model with the age group variable and look at the design matrix:
formula_cat = 'Birthweight ~ ' + \
    ' + '.join(birthweight_new.columns.drop(['Birthweight', 'mage']))
birthweight_mlr_categorical = smf.ols(formula_cat, data=birthweight_new).fit()

# The design matrix shows the dummy variables created for 'mage_group'
design_matrix_cat = pd.DataFrame(
    birthweight_mlr_categorical.model.exog,
    columns=birthweight_mlr_categorical.model.exog_names
)
design_matrix_cat
Out[66]:
Intercept mage_group[T.b] mage_group[T.c] Length Headcirc Gestation smoker mnocig mheight mppwt fage fedyrs fnocig fheight
0 1.0 1.0 0.0 56.0 34.0 44.0 0.0 0.0 162.0 57.0 23.0 10.0 35.0 179.0
1 1.0 0.0 0.0 53.0 36.0 40.0 0.0 0.0 171.0 62.0 19.0 12.0 0.0 183.0
2 1.0 0.0 1.0 58.0 39.0 41.0 0.0 0.0 172.0 58.0 31.0 16.0 25.0 185.0
3 1.0 1.0 0.0 53.0 38.0 44.0 0.0 0.0 174.0 68.0 26.0 14.0 25.0 189.0
4 1.0 1.0 0.0 54.0 37.0 42.0 0.0 0.0 175.0 66.0 30.0 12.0 0.0 184.0
5 1.0 1.0 0.0 51.0 38.0 38.0 0.0 0.0 165.0 61.0 31.0 16.0 0.0 180.0
6 1.0 1.0 0.0 52.0 34.0 40.0 0.0 0.0 157.0 50.0 31.0 16.0 0.0 173.0
7 1.0 1.0 0.0 53.0 33.0 42.0 0.0 0.0 165.0 61.0 21.0 10.0 25.0 185.0
8 1.0 0.0 0.0 54.0 38.0 38.0 0.0 0.0 172.0 50.0 20.0 12.0 7.0 172.0
9 1.0 1.0 0.0 50.0 35.0 38.0 0.0 0.0 157.0 48.0 22.0 14.0 0.0 179.0
10 1.0 1.0 0.0 53.0 33.0 41.0 0.0 0.0 164.0 62.0 37.0 14.0 0.0 170.0
11 1.0 1.0 0.0 51.0 36.0 40.0 0.0 0.0 168.0 53.0 29.0 16.0 0.0 181.0
12 1.0 0.0 1.0 52.0 36.0 38.0 0.0 0.0 164.0 57.0 35.0 16.0 0.0 183.0
13 1.0 1.0 0.0 53.0 33.0 41.0 0.0 0.0 155.0 55.0 25.0 14.0 25.0 183.0
14 1.0 1.0 0.0 53.0 34.0 40.0 0.0 0.0 167.0 60.0 30.0 16.0 25.0 182.0
15 1.0 1.0 0.0 48.0 33.0 37.0 0.0 0.0 158.0 54.0 39.0 10.0 0.0 171.0
16 1.0 1.0 0.0 48.0 35.0 39.0 0.0 0.0 162.0 62.0 27.0 14.0 0.0 178.0
17 1.0 1.0 0.0 48.0 33.0 34.0 0.0 0.0 167.0 64.0 25.0 12.0 25.0 175.0
18 1.0 0.0 0.0 53.0 34.0 39.0 0.0 0.0 165.0 57.0 23.0 14.0 2.0 193.0
19 1.0 1.0 0.0 43.0 32.0 33.0 0.0 0.0 149.0 45.0 26.0 16.0 0.0 169.0
20 1.0 1.0 0.0 53.0 38.0 40.0 1.0 2.0 170.0 59.0 24.0 12.0 12.0 185.0
21 1.0 1.0 0.0 51.0 33.0 41.0 1.0 7.0 160.0 53.0 24.0 16.0 12.0 176.0
22 1.0 0.0 0.0 50.0 30.0 37.0 1.0 7.0 165.0 60.0 20.0 14.0 0.0 183.0
23 1.0 1.0 0.0 50.0 35.0 39.0 1.0 7.0 159.0 52.0 23.0 14.0 25.0 200.0
24 1.0 1.0 0.0 48.0 30.0 37.0 1.0 7.0 163.0 47.0 20.0 10.0 35.0 185.0
25 1.0 0.0 1.0 46.0 32.0 35.0 1.0 7.0 166.0 57.0 37.0 14.0 25.0 173.0
26 1.0 1.0 0.0 48.0 30.0 33.0 1.0 7.0 161.0 50.0 20.0 10.0 35.0 180.0
27 1.0 0.0 1.0 58.0 39.0 41.0 1.0 12.0 173.0 70.0 38.0 14.0 25.0 180.0
28 1.0 0.0 1.0 53.0 34.0 40.0 1.0 12.0 163.0 49.0 41.0 12.0 50.0 191.0
29 1.0 1.0 0.0 49.0 36.0 40.0 1.0 12.0 152.0 48.0 37.0 12.0 25.0 170.0
30 1.0 1.0 0.0 52.0 35.0 38.0 1.0 12.0 165.0 64.0 38.0 14.0 50.0 180.0
31 1.0 1.0 0.0 51.0 38.0 39.0 1.0 17.0 157.0 48.0 32.0 12.0 25.0 169.0
32 1.0 1.0 0.0 50.0 33.0 39.0 1.0 17.0 156.0 53.0 24.0 12.0 7.0 179.0
33 1.0 1.0 0.0 50.0 33.0 45.0 1.0 25.0 163.0 54.0 30.0 16.0 0.0 183.0
34 1.0 1.0 0.0 52.0 36.0 39.0 1.0 25.0 170.0 78.0 40.0 16.0 50.0 178.0
35 1.0 0.0 1.0 53.0 37.0 41.0 1.0 25.0 161.0 66.0 46.0 16.0 0.0 175.0
36 1.0 1.0 0.0 52.0 37.0 40.0 1.0 25.0 170.0 62.0 30.0 10.0 25.0 181.0
37 1.0 1.0 0.0 52.0 33.0 39.0 1.0 25.0 181.0 69.0 23.0 16.0 2.0 181.0
38 1.0 0.0 1.0 49.0 34.0 38.0 1.0 25.0 162.0 57.0 32.0 16.0 50.0 194.0
39 1.0 1.0 0.0 53.0 34.0 41.0 1.0 35.0 163.0 51.0 31.0 16.0 25.0 185.0
40 1.0 1.0 0.0 47.0 33.0 35.0 1.0 35.0 170.0 57.0 23.0 12.0 50.0 186.0
41 1.0 0.0 1.0 53.0 32.0 40.0 1.0 50.0 168.0 61.0 31.0 16.0 0.0 173.0

When fitting the regression model, statsmodels automatically converted the factor to the appropriate number of indicator/dummy variables which were included in the design matrix. We observe that level $a$ is the baseline (it doesn't have its own column) and the two generated columns are for levels $b$ and $c$.

In general, the reference level can be changed as well as the coding. Those changes will affect the estimated $\beta$ coefficients, but they will not affect the fitted or predicted values.


Mean Response Estimation & Prediction¶

In practice, we have target values for which we need to estimate the mean response of values for which we want to do prediction. Since this is not the case here, in order to illustrate both methods, we use as inputs the mean value of every predictor.

In [67]:
# The response variable is 'Birthweight', so we drop it from the mean calculation
mean_values = birthweight.drop('Birthweight', axis=1).mean()

Now, we create the $X$ DataFrame to use as an input in the predict() function:

In [68]:
x_new = pd.DataFrame(mean_values).T

Using the predict function we have:

In [69]:
# Use the predict function
prediction = birthweight_mlr_full.predict(x_new)
prediction
Out[69]:
0    3.312857
dtype: float64

Training vs. Testing Error¶

In this example, and in linear regression models, as we introduce more variables, training error (measured by $\text{RSS}$ or $\text{MSE}$) consistently decreases. However, this reduction in training error does not guarantee a corresponding decrease in test error, which measures prediction accuracy on independent test data. To demonstrate this, we randomly split our dataset into $60\%$ training and $40\%$ test portions, progressively adding predictors.

In [70]:
# Using sklearn's train_test_split is the standard Python approach
X = birthweight.drop('Birthweight', axis=1)
y = birthweight['Birthweight']
p = X.shape[1]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

train_data = pd.concat([X_train, y_train], axis=1)
test_data = pd.concat([X_test, y_test], axis=1)

train_mse = []
test_mse = []
predictor_cols = X.columns.tolist()

for i in range(p):
    current_predictors = predictor_cols[:i + 1]
    formula = 'Birthweight ~ ' + ' + '.join(current_predictors)

    # Fit model on training data
    model = smf.ols(formula, data=train_data).fit()

    # Predict and calculate MSE for training set
    train_pred = model.predict(X_train)
    train_mse.append(np.mean((y_train - train_pred) ** 2))

    # Predict and calculate MSE for test set
    test_pred = model.predict(X_test)
    test_mse.append(np.mean((y_test - test_pred) ** 2))

# Plotting Train vs. Test MSE
plt.figure(figsize=(6, 6))
sns.lineplot(x=range(1, len(predictor_cols) + 1), y=train_mse,
         linestyle='--', marker='s', color='blue', label='Train MSE')
sns.lineplot(x=range(1, len(predictor_cols) + 1), y=test_mse,
         linestyle='-', marker='o' \
         '', color='red', label='Test MSE')
plt.xlabel("# of variables")
plt.ylabel("MSE")
plt.title("Train vs. Test MSE as Predictors are Added")
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)
plt.show()
No description has been provided for this image

In general, we expect that in each iteration, the blue line (representing the training error) consistently exhibits a monotonically decreasing trend, signifying the reduction of training error as more predictors are incorporated. On the other hand, the red line’s (testing error) trajectory may not uniformly decrease.


Test for Outliers¶

To identify outliers, we use the Bonferroni test. For that purpose, we need to compute the studentized residuals $t_{i}$ ( an externally standardized version of residuals ). Under the null hypothesis $H_{0}$, $t_{i} \sim t_{n-p-1}$, where $n$ is the sample size and $p$ is the number of predictors. We perform this test for all $n$ observations testing case at level $\alpha/n$:

In [71]:
# We work with the full model again.
n = birthweight.shape[0]
p = birthweight.shape[1]

We first compute the studentized residuals using the get_influence() function and the Bonferroni critical value using Student’s distribution:

In [72]:
# Computing Studentized Residuals
influence = birthweight_mlr_full.get_influence()
studentized_residuals = influence.resid_studentized_internal

# Critical value WITH Bonferroni correction
alpha = 0.05
bonferroni_cv = sp.stats.t.ppf(alpha / (2 * n), df=n - p - 1)
print(f"Bonferroni critical value: {bonferroni_cv:.4f}")
Bonferroni critical value: -3.6076

Now, we need to find which (if any) studentized residuals exceed the Bonferroni critical value:

In [73]:
# Sorting the residuals in descending order to find outliers (if any)
abs_studentized_resid = np.abs(studentized_residuals)
sorted_resid = np.sort(abs_studentized_resid)[::-1]
print("\nTop 10 absolute studentized residuals:")
print(sorted_resid[:10])

# Find outliers
outliers = sorted_resid[sorted_resid > np.abs(bonferroni_cv)]
print(f"\nNumber of outliers (residuals > critical value): {len(outliers)}")
Top 10 absolute studentized residuals:
[1.96073359 1.81163213 1.76071978 1.63166956 1.59918995 1.57035592
 1.4101976  1.38820038 1.32936059 1.30725325]

Number of outliers (residuals > critical value): 0

Above, we computed a critical t-value of $\lvert-3.6076\rvert$ at $\alpha=0.05$. If an observation's studentized residual is higher (in absolute value) than this critical value, then this observation will be considered an outlier. According to this criterion, we can see that we don’t have any outliers in the data set, since none of the studentized residuals is higher than $3.6076$.


Checking Model Assumptions¶

We start by using residual plots and specifically the residuals against fitted values:

In [74]:
import seaborn as sns

fig, ax = plt.subplots(1, figsize=(6, 6))
fitted = birthweight_mlr_full.fittedvalues
resid = birthweight_mlr_full.resid
sns.residplot(
    x=fitted,
    y=resid,
    lowess=True,
    scatter_kws={'alpha': 0.5},
    line_kws={'color': 'red', 'lw': 1, 'alpha': 0.8},
    ax=ax)

# annotations
residual_abs = np.abs(resid)
abs_resid = np.flip(np.argsort(residual_abs), 0)
abs_resid_top_3 = abs_resid[:3]
for i in abs_resid_top_3:
    ax.annotate(
        i,
        xy=(fitted[i], resid[i]),
        color='C3')

ax.set_title('Residuals vs Fitted', fontweight="bold")
ax.set_xlabel('Fitted values')
ax.set_ylabel('Residuals')
Out[74]:
Text(0, 0.5, 'Residuals')
No description has been provided for this image

The points on the plot are randomly scattered around the zero line, so we conclude that the constant variance assumption is satisfied.

We continue by checking the normality assumption:

In [75]:
fig, ax = plt.subplots(1, 2, figsize=(12, 6))

residual_norm = influence.resid_studentized_internal
QQ = ProbPlot(residual_norm)
fig = QQ.qqplot(line='45', alpha=0.5, lw=1, ax=ax[0])

sns.histplot(resid, stat='density', ax=ax[1]);

ax[0].set_title('Normal Q-Q', fontweight="bold")
ax[0].set_xlabel('Theoretical Quantiles')
ax[0].set_ylabel('Standardized Residuals');

ax[1].set_title('Histogram of Residuals', fontweight="bold")
Out[75]:
Text(0.5, 1.0, 'Histogram of Residuals')
No description has been provided for this image

We seem to have some departures from the normality assumption in this case.


Collinearity: Car Seat Position Data (seatpos)¶

Car designers would find it helpful to know how different drivers will position the seat depending on their size and age. Researchers at the HuMoSim laboratory at the University of Michigan collected the following data on 38 drivers:

  • Age: Drivers age in years
  • Weight: Drivers weight in lbs
  • HtShoes: height with shoes in cm
  • Ht: height without shoes in cm
  • Seated: seated height in cm
  • Arm: lower arm length in cm
  • Thigh: thigh length in cm
  • Leg: lower leg length in cm
  • hipcenter: horizontal distance of the midpoint of the hips from a fixed location in the car in mm

This dataset originates from the faraway R package and a .csv version can be downloaded from Julian Faraway's Linear Models with Python repository.

In [76]:
seatpos = pd.read_csv("seatpos.csv")

# Fit the FULL model
var_names = [x for x in seatpos.columns if x != 'hipcenter']
fml_full = "hipcenter" + "~" + "+".join(x for x in var_names)
position_full = smf.ols(fml_full, data=seatpos).fit()
position_full.summary()
Out[76]:
OLS Regression Results
Dep. Variable: hipcenter R-squared: 0.687
Model: OLS Adj. R-squared: 0.600
Method: Least Squares F-statistic: 7.940
Date: Fri, 21 Aug 2026 Prob (F-statistic): 1.31e-05
Time: 01:31:59 Log-Likelihood: -186.73
No. Observations: 38 AIC: 391.5
Df Residuals: 29 BIC: 406.2
Df Model: 8
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 436.4321 166.572 2.620 0.014 95.755 777.109
Age 0.7757 0.570 1.360 0.184 -0.391 1.942
Weight 0.0263 0.331 0.080 0.937 -0.651 0.703
HtShoes -2.6924 9.753 -0.276 0.784 -22.640 17.255
Ht 0.6013 10.130 0.059 0.953 -20.117 21.319
Seated 0.5338 3.762 0.142 0.888 -7.160 8.228
Arm -1.3281 3.900 -0.341 0.736 -9.305 6.649
Thigh -1.1431 2.660 -0.430 0.671 -6.583 4.297
Leg -6.4390 4.714 -1.366 0.182 -16.080 3.202
Omnibus: 0.543 Durbin-Watson: 1.769
Prob(Omnibus): 0.762 Jarque-Bera (JB): 0.664
Skew: 0.157 Prob(JB): 0.717
Kurtosis: 2.434 Cond. No. 8.44e+03


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 8.44e+03. This might indicate that there are
strong multicollinearity or other numerical problems.

Note that the $p$-value of the overall $F$ test is very small (1.31e-05), which means that at least one of the predictors is statistically significant, but at the same time none of the predictors is statistically significant according to the individual $t$ tests. This is strange and definitely a red flag that we may have collinearity issues.

To investigate the presence of high correlated predictors as well as the linear correlation of the response with each of the predictors, we start by checking the correlation matrix:

In [77]:
seatpos.corr().round(2)
Out[77]:
Age Weight HtShoes Ht Seated Arm Thigh Leg hipcenter
Age 1.00 0.08 -0.08 -0.09 -0.17 0.36 0.09 -0.04 0.21
Weight 0.08 1.00 0.83 0.83 0.78 0.70 0.57 0.78 -0.64
HtShoes -0.08 0.83 1.00 1.00 0.93 0.75 0.72 0.91 -0.80
Ht -0.09 0.83 1.00 1.00 0.93 0.75 0.73 0.91 -0.80
Seated -0.17 0.78 0.93 0.93 1.00 0.63 0.61 0.81 -0.73
Arm 0.36 0.70 0.75 0.75 0.63 1.00 0.67 0.75 -0.59
Thigh 0.09 0.57 0.72 0.73 0.61 0.67 1.00 0.65 -0.59
Leg -0.04 0.78 0.91 0.91 0.81 0.75 0.65 1.00 -0.79
hipcenter 0.21 -0.64 -0.80 -0.80 -0.73 -0.59 -0.59 -0.79 1.00

We observe that the response is highly correlated with most of the variables which justifies the low $p$-value in the overall $F$ test. However, we have highly correlated predictors (e.g. Corr(Leg, HtShoes)=0.91) which means that we will have collinearity.

Let's look at the condition number of the $\mathbf{X}^T\mathbf{X}$ matrix. Recall that the condition number is not scale invariant. This implies that calculating it before standardizing the matrix might lead to incorrect conclusions. So, let's standardize it first:

In [78]:
# Extract the design matrix and remove the column of 1s that corresponds to the intercept.
X = position_full.model.exog[:, 1:]

# Standardize the matrix
scaler = StandardScaler()
X = scaler.fit_transform(X)

# show the variance
X.var(axis=0, ddof=0)
Out[78]:
array([1., 1., 1., 1., 1., 1., 1., 1.])

Compute the condition number using the standardized matrix:

In [79]:
# Compute the eigenvalues of the correlation matrix (X^T*X for standardized data)
eigenvalues, _ = np.linalg.eig(X.T @ X)

print("Eigenvalues of the correlation matrix:")
print(np.sort(eigenvalues)[::-1])

# Compute the condition number:
condition_number = np.sqrt(eigenvalues.max() / eigenvalues.min())
print(f"Condition Number: {condition_number:.4f}")
Eigenvalues of the correlation matrix:
[2.15581055e+02 4.69978684e+01 1.76222508e+01 9.15641723e+00
 7.38034342e+00 5.28861961e+00 1.91309203e+00 6.03530674e-02]
Condition Number: 59.7662

The condition number is 59.77, larger than 30, so we conclude that collinearity is present.

Let's also check the Variance Inflation Factor (VIF):

In [80]:
variance_inflation_factor(X, 1)
Out[80]:
np.float64(3.647030128812391)
In [81]:
p = X.shape[1]
vif_data = np.array([variance_inflation_factor(X, i) for i in range(p)])
display(vif_data)

np.sqrt(307.43)
array([  1.99793148,   3.64703013, 307.42937802, 333.13783239,
         8.9510538 ,   4.49636844,   2.76288552,   6.69429122])
Out[81]:
np.float64(17.5336818723279)

Note that the standard error for the coefficient associated with HtShoes is 17.53 times larger than it would have been without collinearity.

The next step is to investigate which variable to remove. To do so, we study the pairwise correlations and perform partial $F$-tests as follows:

In [82]:
position_red1 = smf.ols(
    'hipcenter ~ Age + Weight + Ht + Seated', data=seatpos).fit()

position_red1.summary()
Out[82]:
OLS Regression Results
Dep. Variable: hipcenter R-squared: 0.660
Model: OLS Adj. R-squared: 0.619
Method: Least Squares F-statistic: 16.01
Date: Fri, 21 Aug 2026 Prob (F-statistic): 2.22e-07
Time: 01:31:59 Log-Likelihood: -188.28
No. Observations: 38 AIC: 386.6
Df Residuals: 33 BIC: 394.8
Df Model: 4
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 478.6589 159.734 2.997 0.005 153.678 803.639
Age 0.5840 0.426 1.372 0.179 -0.282 1.450
Weight -0.0154 0.316 -0.049 0.962 -0.659 0.628
Ht -4.9902 1.644 -3.036 0.005 -8.335 -1.646
Seated 2.0463 3.413 0.600 0.553 -4.897 8.990
Omnibus: 0.512 Durbin-Watson: 1.672
Prob(Omnibus): 0.774 Jarque-Bera (JB): 0.435
Skew: -0.244 Prob(JB): 0.805
Kurtosis: 2.810 Cond. No. 6.70e+03


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 6.7e+03. This might indicate that there are
strong multicollinearity or other numerical problems.
In [83]:
position_red2 = smf.ols('hipcenter ~ Ht', data=seatpos).fit()
display(position_red2.summary())
OLS Regression Results
Dep. Variable: hipcenter R-squared: 0.638
Model: OLS Adj. R-squared: 0.628
Method: Least Squares F-statistic: 63.53
Date: Fri, 21 Aug 2026 Prob (F-statistic): 1.83e-09
Time: 01:31:59 Log-Likelihood: -189.45
No. Observations: 38 AIC: 382.9
Df Residuals: 36 BIC: 386.2
Df Model: 1
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 556.2553 90.670 6.135 0.000 372.367 740.144
Ht -4.2650 0.535 -7.970 0.000 -5.350 -3.180
Omnibus: 1.707 Durbin-Watson: 1.890
Prob(Omnibus): 0.426 Jarque-Bera (JB): 1.028
Skew: -0.395 Prob(JB): 0.598
Kurtosis: 3.154 Cond. No. 2.60e+03


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 2.6e+03. This might indicate that there are
strong multicollinearity or other numerical problems.
In [84]:
anova_seatpos = sm.stats.anova_lm(position_red2, position_red1)
anova_seatpos
Out[84]:
df_resid ssr df_diff ss_diff F Pr(>F)
0 36.0 47615.793248 0.0 NaN NaN NaN
1 33.0 44774.245915 3.0 2841.547333 0.698103 0.559892

Based on the $F$ test provided in the ANOVA table, we conclude that the reduced model with Ht as the only variable is better than the model that includes Age, Weight, Ht and Seated.