WWS509 - Lecture Notes

B.3 Tests of Hypotheses

We consider Wald tests and likelihood ratio tests, introducing the deviance statistic.

B.3.1 Wald Tests

The Wald test follows immediately from the fact that the information matrix for generalized linear models is given by

I(b) = XWX/f,

(B.9)

so the large sample distribution of the maximum likelihood estimator [^(b)] is multivariate normal

^
b

~ N_p( b, (XWX)^-1f).

(B.10)

with mean b and variance-covariance matrix (XWX)^-1f.

Tests for subsets of b are based on the corresponding marginal normal distributions.

Example: In the case of normal errors with identity link we have W = I (where I denotes the identity matrix), f = s², and the exact distribution of [^(b)] is multivariate normal with mean b and variance-covariance matrix (XX)^-1s².

B.3.2 Likelihood Ratio Tests and The Deviance

We will show how the likelihood ratio criterion for comparing any two nested models, say w₁ w₂, can be constructed in terms of a statistic called the deviance and an unknown scale parameter f.

Consider first comparing a model of interest w with a saturated model W that provides a separate parameter for each observation.

Let [^(m)]_i denote the fitted values under w and let [^(q)]_i denote the corresponding estimates of the canonical parameters. Similarly, let [(m)\tilde]_O = y_i and [(q)\tilde]_i denote the corresponding estimates under W.

The likelihood ratio criterion to compare these two models in the exponential family has the form

-2logl = 2

n

i = 1

y_i(

~
q_i

^
q_i

)-b(

~
q_i

) + b(

^
q_i

)

a_i(f)

Assume as usual that a_i(f) = f/p_i for known prior weights p_i. Then we can write the likelihood-ratio criterion as follows:

-2logl =

D(y,

^
m

)

(B.11)

The numerator of this expression does not depend on unknown parameters and is called the deviance:

D(y,

^
m

) = 2

n

i = 1

p_i [ y_i(

~
q_i

^
q_i

)- b(

~
q_i

) + b(

^
q_i

) ].

(B.12)

The likelihood ratio criterion -2logL is the deviance divided by the scale parameter f, and is called the scaled deviance.

Example: Recall that for the normal distribution we had q_i = m_i, b(q_i) = [1/2]q_i², and a_i(f) = s², so the prior weights are p_i = 1. Thus, the deviance is

D(y,

^
m

)

{ y_i(y_i-

^
m_i

y_i² +

^
m_i

}

{

y_i² - y_i

^
m_i

}

(y_i -

^
m_i

)²

our good old friend, the residual sum of squares. ^[¯]

Let us now return to the comparison of two nested models w₁, with p₁ parameters, and w₂, with p₂ parameters, such that w₁ w₂ and p₂ > p1.

The log of the ratio of maximized likelihoods under the two models can be written as a difference of deviances, since the maximized log-likelihood under the saturated model cancels out. Thus, we have

-2logl =

D(w₁)-D(w₂)

(B.13)

The scale parameter f is either known or estimated using the larger model w₂.

Large sample theory tells us that the asymptotic distribution of this criterion under the usual regularity conditions is c²_n with n = p₂-p₁ degrees of freedom.

Example: In the linear model with normal errors we estimate the unknown scale parameter f using the residual sum of squares of the larger model, so the criterion becomes

-2logl =

RSS(w₁)-RSS(w₂)

RSS(w₂)/(n-p₂)

In large samples the approximate distribution of this criterion is c²_n with n = p₂-p₁ degrees of freedom. Under normality, however, we have an exact result: dividing the criterion by p₂-p₁ we obtain an F with p₂-p₁ and n-p₂ degrees of freedom. Note that as n the degrees of freedom in the denominator approach and the F converges to (p₂-p₁)c², so the asymptotic and exact criteria become equivalent. ^[¯]

In Sections B.4 and B.5 we will construct likelihood ratio tests for binomial and Poisson data. In those cases f = 1 (unless one allows over-dispersion and estimates f, but that's another story) and the deviance is the same as the scaled deviance. All our tests will be based on asymptotic c² statistics.