8.4 Logistic Regression for the Heart Data Set in R

The Python version of the example can be found here: Python Code for Logistic . 

In this example we work with the South African heart data, available here.

SAheart = read.csv("data/week8/SAheart.csv", header = TRUE)

The SAheart data set comes from a study of heart-disease risk factors among South African men of mixed ancestry. It contains \(462\) observations on lifestyle, medical history, and physiological measurements—age, alcohol intake, tobacco use, blood pressure, cholesterol, and family history of heart disease, among others.

Our goal is to estimate the probability that an individual has coronary heart disease (CHD).

The variables are:

  • sbp - systolic blood pressure

  • tobacco - cumulative tobacco (kg)

  • ldl - low-density lipoprotein cholesterol

  • adiposity - a numeric measure of relative body fat

  • famhist - family history of heart disease (Present, Absent)

  • typea - Type A behavior

  • obesity

  • alcohol - current alcohol consumption

  • age - age at onset

  • chd - response: coronary heart disease (1 = yes, 0 = no)

heart.logistic = glm(chd ~ ., data = SAheart, family = binomial)
summary(heart.logistic)
## 
## Call:
## glm(formula = chd ~ ., family = binomial, data = SAheart)
## 
## Coefficients:
##                  Estimate Std. Error z value Pr(>|z|)    
## (Intercept)    -5.9207616  1.3265724  -4.463 8.07e-06 ***
## row.names      -0.0008844  0.0008950  -0.988 0.323042    
## sbp             0.0076602  0.0058574   1.308 0.190942    
## tobacco         0.0777962  0.0266602   2.918 0.003522 ** 
## ldl             0.1701708  0.0597998   2.846 0.004432 ** 
## adiposity       0.0209609  0.0294496   0.712 0.476617    
## famhistPresent  0.9385467  0.2287202   4.103 4.07e-05 ***
## typea           0.0376529  0.0124706   3.019 0.002533 ** 
## obesity        -0.0661926  0.0443180  -1.494 0.135285    
## alcohol         0.0004222  0.0045053   0.094 0.925346    
## age             0.0441808  0.0121784   3.628 0.000286 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 596.11  on 461  degrees of freedom
## Residual deviance: 471.16  on 451  degrees of freedom
## AIC: 493.16
## 
## Number of Fisher Scoring iterations: 5

We obtain fitted values from the model and summarize the classifications at the \(0.5\) threshold:

yhat = (heart.logistic$fitted.values > 0.5)
table(yhat, SAheart$chd)
##        
## yhat      0   1
##   FALSE 255  79
##   TRUE   47  81

8.4.1 Goodness of Fit

In logistic regression we do not have a residual sum of squares (RSS), so we need other metrics to assess fit. Deviance measures, which are also based on the likelihood, are the standard choice. When all of the \(x_i\) are unique, the residual deviance is \[ \mathrm{Deviance} = -2 \Biggl( \sum_{i: y_i=1} \log \hat{p}_i + \sum_{i: y_i=0} \log(1-\hat{p}_i) \Biggr), \] where \(\hat{p}_i\) is the fitted probability \(P(Y=1 \mid X=x_i)\).

At the bottom of the summary output in R we see two deviances:

  • Null deviance: the deviance of a logistic regression with only an intercept. Here the estimated probability \(P(Y=1 \mid X=x_i)\) is the sample mean of the observed \(y_i\) values and does not depend on \(x_i\).

  • Residual deviance: the deviance of the fitted model under consideration, using the model’s fitted probabilities \(\hat{p}_i\).

We can also compute them directly:

p.hat = heart.logistic$fitted.values

-2 * (sum(log(p.hat[SAheart$chd == 1])) + sum(log(1 - p.hat[SAheart$chd ==
    0])))
## [1] 471.1591
p0.hat = rep(mean(SAheart$chd), length(SAheart$chd))

-2 * (sum(log(p0.hat[SAheart$chd == 1])) + sum(log(1 - p0.hat[SAheart$chd ==
    0])))
## [1] 596.1084

Solving this problem using numerical optimization, without glm

## Log-likelihood (not the negative log-likelihood)
logistic.loglik <- function(b, x, y) {
    bm = as.matrix(b)
    xb = x %*% bm
    # this returns the log-likelihood
    return(sum(y * xb) - sum(log(1 + exp(xb))))
}

## Gradient
logistic.gradient <- function(b, x, y) {
    bm = as.matrix(b)
    expxb = exp(x %*% bm)
    return(t(x) %*% (y - expxb/(1 + expxb)))
}

## We can now use the `optim` function to solve the
## problem:

SAheart$famhist = ifelse(SAheart$famhist == "Present", 1, 0)

x = as.matrix(cbind(intercept = 1, SAheart[, 2:10]))
y = as.matrix(SAheart[, 11])
b = rep(0, ncol(x))

# The objective above is the log-likelihood, so we maximize
# (equivalently, optim minimizes -ell when fnscale = -1):

optim(b, fn = logistic.loglik, gr = logistic.gradient, method = "BFGS",
    x = x, y = y, control = list(fnscale = -1))
## $par
##  [1] -6.150733305  0.006504017  0.079376464  0.173923988  0.018586578  0.925372019
##  [7]  0.039595096 -0.062909867  0.000121675  0.045225500
## 
## $value
## [1] -236.07
## 
## $counts
## function gradient 
##       74       16 
## 
## $convergence
## [1] 0
## 
## $message
## NULL

The result matches the glm() solution. If a general-purpose solver is not available, we can use Newton–Raphson instead; that requires a function for the Hessian.

  

8.4.2 Variable Selection

Variable selection can also be needed in a logistic regression setting. We next ask which of the methods and criteria from linear regression carry over.

AIC/BIC

We can still use AIC and BIC because the logistic model has a likelihood. Unlike linear regression, however, we cannot use the leaps-and-bounds algorithm to enumerate all submodels, because the logistic likelihood is not a residual sum of squares. Instead we use a greedy search: stepwise, backward, or forward selection.

As an example:

step.logistic = step(heart.logistic, scope = list(upper = ~.,
    lower = ~1))
## Start:  AIC=493.16
## chd ~ row.names + sbp + tobacco + ldl + adiposity + famhist + 
##     typea + obesity + alcohol + age
## 
##             Df Deviance    AIC
## - alcohol    1   471.17 491.17
## - adiposity  1   471.67 491.67
## - row.names  1   472.14 492.14
## - sbp        1   472.88 492.88
## <none>           471.16 493.16
## - obesity    1   473.47 493.47
## - ldl        1   479.65 499.65
## - tobacco    1   480.27 500.27
## - typea      1   480.75 500.75
## - age        1   484.76 504.76
## - famhist    1   488.29 508.29
## 
## Step:  AIC=491.17
## chd ~ row.names + sbp + tobacco + ldl + adiposity + famhist + 
##     typea + obesity + age
## 
##             Df Deviance    AIC
## - adiposity  1   471.69 489.69
## - row.names  1   472.14 490.14
## - sbp        1   472.94 490.94
## <none>           471.17 491.17
## - obesity    1   473.49 491.49
## + alcohol    1   471.16 493.16
## - ldl        1   479.68 497.68
## - tobacco    1   480.73 498.73
## - typea      1   480.80 498.80
## - age        1   484.82 502.82
## - famhist    1   488.43 506.43
## 
## Step:  AIC=489.69
## chd ~ row.names + sbp + tobacco + ldl + famhist + typea + obesity + 
##     age
## 
##             Df Deviance    AIC
## - row.names  1   472.55 488.55
## - sbp        1   473.55 489.55
## <none>           471.69 489.69
## - obesity    1   473.86 489.86
## + adiposity  1   471.17 491.17
## + alcohol    1   471.67 491.67
## - typea      1   481.05 497.05
## - tobacco    1   481.35 497.35
## - ldl        1   481.68 497.68
## - famhist    1   488.89 504.89
## - age        1   493.97 509.97
## 
## Step:  AIC=488.55
## chd ~ sbp + tobacco + ldl + famhist + typea + obesity + age
## 
##             Df Deviance    AIC
## - sbp        1   473.98 487.98
## <none>           472.55 488.55
## - obesity    1   474.65 488.65
## + row.names  1   471.69 489.69
## + adiposity  1   472.14 490.14
## + alcohol    1   472.55 490.55
## - tobacco    1   482.54 496.54
## - ldl        1   482.95 496.95
## - typea      1   483.19 497.19
## - famhist    1   489.38 503.38
## - age        1   495.48 509.48
## 
## Step:  AIC=487.98
## chd ~ tobacco + ldl + famhist + typea + obesity + age
## 
##             Df Deviance    AIC
## - obesity    1   475.69 487.69
## <none>           473.98 487.98
## + sbp        1   472.55 488.55
## + adiposity  1   473.47 489.47
## + row.names  1   473.55 489.55
## + alcohol    1   473.94 489.94
## - tobacco    1   484.18 496.18
## - typea      1   484.30 496.30
## - ldl        1   484.53 496.53
## - famhist    1   490.58 502.58
## - age        1   502.11 514.11
## 
## Step:  AIC=487.69
## chd ~ tobacco + ldl + famhist + typea + age
## 
##             Df Deviance    AIC
## <none>           475.69 487.69
## + obesity    1   473.98 487.98
## + sbp        1   474.65 488.65
## + row.names  1   475.26 489.26
## + adiposity  1   475.44 489.44
## + alcohol    1   475.65 489.65
## - ldl        1   484.71 494.71
## - typea      1   485.44 495.44
## - tobacco    1   486.03 496.03
## - famhist    1   492.09 502.09
## - age        1   502.38 512.38

8.4.3 Penalized Logistic Regression

As in ordinary regression, we can penalize the logistic log-likelihood to handle collinearity or high-dimensional settings. In R this is again available through the glmnet package.

library(glmnet)
## Loading required package: Matrix
## Loaded glmnet 5.0
lasso.fit = cv.glmnet(x = data.matrix(SAheart[, 2:10]), y = SAheart[,
    11], nfolds = 10, family = "binomial")

plot(lasso.fit)

coef(lasso.fit, s = "lambda.min")
## 10 x 1 sparse Matrix of class "dgCMatrix"
##              lambda.min
## (Intercept) -5.73717333
## sbp          0.00416794
## tobacco      0.07056290
## ldl          0.14789200
## adiposity    .         
## famhist      0.81077289
## typea        0.02967640
## obesity     -0.01618876
## alcohol      .         
## age          0.04396417

8.4.4 Illustration of Divergence of the MLE

Consider a very simple case in which \[y = \begin{cases} 1, & \text{if } x_1 - 2 x_2 > 0,\\ 0, & \text{otherwise.} \end{cases} \]

Simulate such data:

x = matrix(runif(100), 50, 2)
y = ifelse(x[, 1] - 2 * x[, 2] > 0, 1, 0)
plot(x[, 1], x[, 2], type = "n", xlab = "", ylab = "")
text(x[, 1], x[, 2], y)

logistic.fit = glm(y ~ x, family = binomial)
## Warning: glm.fit: algorithm did not converge
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred

The warning is exactly the phenomenon discussed above: the MLE of the coefficients tends to \(\pm\infty\) and the algorithm does not converge.

Even so, the separating line is well defined, so the fitted model can still be used for prediction.

round(logistic.fit$fitted.values, 0)
##  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
##  0  0  0  1  1  0  1  0  1  0  1  1  0  0  0  0  0  0  1  0  0  0  1  0  0  0  0  0  0  0  0 
## 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 
##  1  0  1  1  1  0  1  0  0  0  0  0  0  0  0  0  1  0  0