8.4b Logistic Regression for the Heart Data Set in Python¶

In [ ]:
%matplotlib inline

import pandas as pd
import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt
from IPython.display import display
from statsmodels.api import Logit, add_constant
from scipy.optimize import minimize
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler

In this example we work with the South African heart data, available here.

In [ ]:
url = "https://web.stanford.edu/~hastie/ElemStatLearn/datasets/SAheart.data"
SAheart = pd.read_csv(url, index_col=0)
SAheart.head()

The SAheart data set comes from a study of heart-disease risk factors among South African men of mixed ancestry. It contains $462$ observations on lifestyle, medical history, and physiological measurements---age, alcohol intake, tobacco use, blood pressure, cholesterol, and family history of heart disease, among others.

Our goal is to estimate the probability that an individual has coronary heart disease (CHD).

The variables are:

  • sbp - systolic blood pressure

  • tobacco - cumulative tobacco (kg)

  • ldl - low-density lipoprotein cholesterol

  • adiposity - a numeric measure of relative body fat

  • famhist - family history of heart disease (Present, Absent)

  • typea - Type A behavior

  • obesity

  • alcohol - current alcohol consumption

  • age - age at onset

  • chd - response: coronary heart disease (1 = yes, 0 = no)

statsmodels expects a numeric design matrix, so we recode famhist and add an intercept column (the analog of glm(chd ~ ., family = binomial) in R).

In [ ]:
SAheart_encoded = SAheart.copy()
SAheart_encoded["famhist"] = (SAheart_encoded["famhist"] == "Present").astype(int)

X_features = SAheart_encoded.drop("chd", axis=1)
X = add_constant(X_features, has_constant="add")
y = SAheart_encoded["chd"]

heart_logistic = Logit(y, X).fit()
heart_logistic.summary()

We obtain fitted values from the model and summarize the classifications at the $0.5$ threshold:

In [ ]:
yhat = (heart_logistic.predict(X) > 0.5).astype(int)
pd.crosstab(yhat, y, rownames=["Predicted"], colnames=["Actual"])

Goodness of Fit¶

In logistic regression we do not have a residual sum of squares (RSS), so we need other metrics to assess fit. Deviance measures, which are also based on the likelihood, are the standard choice. When all of the $x_i$ are unique, the residual deviance is

$$ \mathrm{Deviance} = -2 \Biggl( \sum_{i: y_i=1} \log \hat{p}_i + \sum_{i: y_i=0} \log(1-\hat{p}_i) \Biggr), $$

where $\hat{p}_i$ is the fitted probability $P(Y=1 \mid X=x_i)$.

statsmodels reports the log-likelihood of the fitted model and of the intercept-only model (LL-Null). Multiplying by $-2$ recovers the two deviances that R prints at the bottom of summary.glm:

  • Null deviance: the deviance of a logistic regression with only an intercept. Here the estimated probability $P(Y=1 \mid X=x_i)$ is the sample mean of the observed $y_i$ values and does not depend on $x_i$.

  • Residual deviance: the deviance of the fitted model under consideration, using the model's fitted probabilities $\hat{p}_i$.

We can also compute them directly:

In [ ]:
p_hat = heart_logistic.predict(X)
-2 * (np.sum(np.log(p_hat[y == 1])) + np.sum(np.log(1 - p_hat[y == 0])))
In [ ]:
p0_hat = np.full(len(y), y.mean())
-2 * (np.sum(np.log(p0_hat[y == 1])) + np.sum(np.log(1 - p0_hat[y == 0])))

Check against the statsmodels log-likelihoods:

In [ ]:
print(-2 * heart_logistic.llf)      # residual deviance
print(-2 * heart_logistic.llnull)   # null deviance

Numerical optimization without Logit¶

The functions below are the same log-likelihood and gradient used in the R notes. scipy.optimize.minimize minimizes, so we pass the negative log-likelihood (R's optim maximized with fnscale = -1).

In [ ]:
def logistic_loglik(b, x, y):
    xb = x @ b
    return np.sum(y * xb) - np.sum(np.log(1 + np.exp(xb)))

def logistic_gradient(b, x, y):
    expxb = np.exp(x @ b)
    return x.T @ (y - expxb / (1 + expxb))

x_mat = X.values
y_vec = y.values.astype(float)
b_init = np.zeros(x_mat.shape[1])

result = minimize(
    fun=lambda b: -logistic_loglik(b, x_mat, y_vec),
    x0=b_init,
    method="BFGS",
    jac=lambda b: -logistic_gradient(b, x_mat, y_vec),
)

pd.Series(result.x, index=X.columns, name="coef")

The result matches the statsmodels / glm() solution. If a general-purpose solver is not available, we can use Newton--Raphson instead; that requires a function for the Hessian.

Variable Selection¶

Variable selection can also be needed in a logistic regression setting. We next ask which of the methods and criteria from linear regression carry over.

AIC / BIC¶

We can still use AIC and BIC because the logistic model has a likelihood. Unlike linear regression, however, we cannot use the leaps-and-bounds algorithm to enumerate all submodels, because the logistic likelihood is not a residual sum of squares. Instead we use a greedy search: stepwise, backward, or forward selection.

R's step() starts from the fitted full model and uses AIC. The function below does the same kind of both-direction AIC search. It operates on the predictors only; the intercept is always included, as in glm.

In [ ]:
def stepwise_logistic(X_pred, y, start="full"):
    """AIC stepwise selection (forward / backward / both).

    X_pred should not contain an intercept column.
    start='full' matches R's step() on the fitted full glm.
    """
    all_vars = list(X_pred.columns)
    included = list(all_vars) if start == "full" else []

    def fit_on(cols):
        return sm.Logit(y, add_constant(X_pred[cols], has_constant="add")).fit(disp=0)

    while True:
        changed = False
        current_aic = fit_on(included).aic

        excluded = [v for v in all_vars if v not in included]
        if excluded:
            add_aic = pd.Series({v: fit_on(included + [v]).aic for v in excluded})
            if add_aic.min() < current_aic:
                included.append(add_aic.idxmin())
                current_aic = add_aic.min()
                changed = True

        if included:
            drop_aic = pd.Series(
                {v: fit_on([u for u in included if u != v]).aic for v in included}
            )
            if drop_aic.min() < current_aic:
                included.remove(drop_aic.idxmin())
                changed = True

        if not changed:
            break

    return fit_on(included)
In [ ]:
final_model = stepwise_logistic(X_features, y, start="full")
final_model.summary()

Penalized Logistic Regression¶

As in ordinary regression, we can penalize the logistic log-likelihood to handle collinearity or high-dimensional settings. In R this is done with cv.glmnet(..., family = "binomial").

In Python we cross-validate an L1-penalized logistic regression over a grid of $C = 1/\lambda$ values. Predictors are standardized first, matching glmnet's default. The selected coefficients will not match glmnet exactly, but the exercise is the same.

In [ ]:
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_features)

Cs = np.logspace(-4, 4, 30)
lasso_cv = GridSearchCV(
    LogisticRegression(
        l1_ratio=1.0,      # L1 penalty (sklearn >= 1.8)
        solver="saga",
        max_iter=10000,
        random_state=0,
    ),
    param_grid={"C": Cs},
    scoring="neg_log_loss",
    cv=10,
)
lasso_cv.fit(X_scaled, y)

mean_deviance = -lasso_cv.cv_results_["mean_test_score"]
plt.figure(figsize=(6, 4))
plt.semilogx(Cs, mean_deviance, marker="o")
plt.axvline(lasso_cv.best_params_["C"], linestyle="--")
plt.xlabel("C (1 / lambda)")
plt.ylabel("Mean binomial deviance (CV)")
plt.title("L1-penalized logistic regression path")
plt.show()

print("best C:", lasso_cv.best_params_["C"])
pd.Series(lasso_cv.best_estimator_.coef_[0], index=X_features.columns, name="coef")

Illustration of Divergence of the MLE¶

Consider a very simple case in which

$$ y = \begin{cases} 1, & \text{if } x_1 - 2 x_2 > 0, \\ 0, & \text{otherwise.} \end{cases} $$

Simulate such data:

In [ ]:
rng = np.random.default_rng(42)
x = rng.uniform(0, 1, size=(50, 2))
y_sep = (x[:, 0] - 2 * x[:, 1] > 0).astype(int)

plt.figure(figsize=(6, 5))
plt.scatter(x[:, 0], x[:, 1], c="white")
for i in range(len(x)):
    plt.text(x[i, 0], x[i, 1], str(y_sep[i]), ha="center", va="center")
plt.xlabel("x1")
plt.ylabel("x2")
plt.show()
In [ ]:
X_sep = add_constant(x, has_constant="add")
logistic_sep = Logit(y_sep, X_sep).fit()
np.round(logistic_sep.predict(X_sep), 0)

The warning is exactly the phenomenon discussed in the notes: the MLE of the coefficients tends to $\pm\infty$ and the algorithm does not converge.

Even so, the separating line is well defined, so the fitted model can still be used for prediction (the rounded fitted values recover the labels).