WWS509 - Lecture Notes

B.1 The Model

Let y₁, , y_n denote n independent observations on a response. We treat y_i as a realization of a random variable Y_i. In the general linear model we assume that Y_i has a normal distribution with mean m_i and variance s²

Y_i ~ N(m_i, s²),

and we further assume that the expected value m_i is a linear function of p predictors that take values x_i = (x_i1, , x_ip) for the i-th case, so that

m_i = x_ib,

where b is a vector of unknown parameters.

We will generalize this in two steps, dealing with the stochastic and systematic components of the model.

B.1.1 The Exponential Family

We will assume that the observations come from a distribution in the exponential family with probability density function

f(y_i) = exp{

y_i q_i - b(q_i)

a_i(f)

+c(y_i, f) }.

(B.1)

Here q_i and f are parameters and a_i(f), b(q_i) and c(y_i, f) are known functions. In all models considered in these notes the function a_i(f) has the form

a_i(f) = f/p_i,

where p_i is a known prior weight, usually 1.

The parameters q_i and f are essentially location and scale parameters. It can be shown that if Y_i has a distribution in the exponential family then it has mean and variance

E(Y_i)

m_i = b(q_i)

(B.2)

var(Y_i)

s²_i = b(q_i) a_i(f),

(B.3)

where b(q_i) and b(q_i) are the first and second derivatives of b(q_i). When a_i(f) = f/p_i the variance has the simpler form

var(Y_i) = s²_i = fb(q_i)/p_i.

The exponential family just defined includes as special cases the normal, binomial, Poisson, exponential, gamma and inverse Gaussian distributions.

Example: The normal distribution has density

f(y_i) =

2ps²

exp{-

(y_i-m_i)²

s²

Expanding the square in the exponent we get (y_i-m_i)² = y_i² + m_i² - 2 y_i m_i, so the coefficient of y_i is m_i/s². This result identifies q_i as m_i and f as s², with a_i(f) = f. Now write

f(y_i) = exp{

y_i m_i-

m_i²

s²

y_i²

2s²

log(2ps²)}.

This shows that b(q_i) = [1/2]q_i² (recall that q_i = m_i). Let us check the mean and variance:

E(Y_i) = b(q_i) = q_i = m_i,

var(Y_i) = b(q_i)a_i(f) = s².

Try to generalize this result to the case where Y_i has a normal distribution with mean m_i and variance s²/n_i for known constants n_i, as would be the case if the Y_i represented sample means. ^[¯]

Example: In Problem Set 1 you will show that the exponential distribution with density

f(y_i) = l_i exp{ -l_i y_i}

belongs to the exponential family. ^[¯]

In Sections B.4 and B.5 we verify that the binomial and Poisson distributions also belong to this family.

B.1.2 The Link Function

The second element of the generalization is that instead of modeling the mean, as before, we will introduce a one-to-one continuous differentiable transformation g(m_i) and focus on

h_i = g(m_i).

(B.4)

The function g(m_i) will be called the link function. Examples of link functions include the identity, log, reciprocal, logit and probit.

We further assume that the transformed mean follows a linear model, so that

h_i = x_ib.

(B.5)

The quantity h_i is called the linear predictor. Note that the model for h_i is pleasantly simple. Since the link function is one-to-one we can invert it to obtain

m_i = g^-1(x_ib).

The model for m_i is usually more complicated than the model for h_i.

Note that we do not transform the response y_i, but rather its expected value m_i. A model where logy_i is linear on x_i, for example, is not the same as a generalized linear model where logm_i is linear on x_i.

Example: The standard linear model we have studied so far can be described as a generalized linear model with normal errors and identity link, so that

h_i = m_i.

It also happens that m_i, and therefore h_i, is the same as q_i, the parameter in the exponential family density. ^[¯]

When the link function makes the linear predictor h_i the same as the canonical parameter q_i, we say that we have a canonical link . The identity is the canonical link for the normal distribution. In later sections we will see that the logit is the canonical link for the binomial distribution and the log is the canonical link for the Poisson distribution. This leads to some natural pairings:

Error	Link
Normal	Identity
Binomial	Logit
Poisson	Log

However, other combinations are also possible. An advantage of canonical links is that a minimal sufficient statistic for b exists, i.e. all the information about b is contained in a function of the data of the same dimensionality as b.