WWS509 - Lecture Notes

A.1 Maximum Likelihood Estimation

Let Y₁, , Y_n be n independent random variables with probability density functions (pdfs) f_i(y_i;q) depending on a vector-valued parameter q.

A.1.1 The Log-likelihood Function

The joint density of n independent observations y = (y₁, , y_n) is

f(y;q) =

n

i = 1

f_i(y_i; q) = L(q;y).

(A.1)

This expression, viewed as a function of the unknown parameter q given the data y, is called the likelihood function.

Often we work with the natural logarithm of the likelihood function, the so-called log-likelihood function:

logL(q;y) =

n

i = 1

logf_i(y_i; q).

(A.2)

A sensible way to estimate the parameter q given the data y is to maximize the likelihood (or equivalently the log-likelihood) function, choosing the parameter value that makes the data actually observed as likely as possible. Formally, we define the maximum-likelihood estimator (mle) as the value [^(q)] such that

logL(

^
q

;y) logL(q;y) for all q.

(A.3)

Example: The Log-Likelihood for the Geometric Distribution. Consider a series of independent Bernoulli trials with common probability of success p. The distribution of the number of failures Y_i before the first success has pdf

Pr(Y_i = y_i) = (1-p)^y_i p.

(A.4)

for y_i = 0, 1, . Direct calculation shows that E(Y_i) = (1-p)/p.

The log-likelihood function based on n observations y can be written as

logL(p;y)

n

i = 1

{ y_ilog(1-p)+logp}

(A.5)

n (

_
y

log(1-p) + logp),

(A.6)

where [`y] = y_i/n is the sample mean. The fact that the log-likelihood depends on the observations only through the sample mean shows that [`y] is a sufficient statistic for the unknown probability p.

Figure A.1: The Geometric Log-Likelihood for n = 20 and [`y] = 3

Figure A.1 shows the log-likelihood function for a sample of n = 20 observations from a geometric distribution when the observed sample mean is [`y] = 3. ^[¯]

A.1.2 The Score Vector

The first derivative of the log-likelihood function is called Fisher's score function, and is denoted by

u(q) =

logL(q;y)

(A.7)

Note that the score is a vector of first partial derivatives, one for each element of q.

If the log-likelihood is concave, one can find the maximum likelihood estimator by setting the score to zero, i.e. by solving the system of equations:

^
q

) = 0.

(A.8)

Example: The Score Function for the Geometric Distribution. The score function for n observations from a geometric distribution is

u(p) =

dlogL

= n (

_
y

1-p

(A.9)

Setting this equation to zero and solving for p leads to the maximum likelihood estimator

^
p

_
y

(A.10)

Note that the mle of the probability of success is the reciprocal of the number of trials. This result is intuitively reasonable: the longer it takes to get a success, the lower our estimate of the probability of success would be.

Suppose now that in a sample of n = 20 observations we have obtained a sample mean of [`y] = 3. The mle of the probability of success would be [^(p)] = 1/(1+3) = 0.25, and it should be clear from Figure A.1 that this value maximizes the log-likelihood. ^[¯]

A.1.3 The Information Matrix

The score is a random vector with some interesting statistical properties. In particular, the score evaluated at the true parameter value q has mean zero

E[u(q)] = 0

and variance-covariance matrix given by the information matrix:

var[u(q)] = E[u(q)u(q)] = I(q).

(A.11)

Under mild regularity conditions, the information matrix can also be obtained as minus the expected value of the second derivatives of the log-likelihood:

I(q) = - E[

²logL(q)

(A.12)

The matrix of negative observed second derivatives is sometimes called the observed information matrix.

Note that the second derivative indicates the extent to which the log-likelihood function is peaked rather than flat. This makes the interpretation in terms of information intuitively reasonable.

Example: Information for the Geometric Distribution. Differentiating the score we find the observed information to be

d²logL

dp²

= -

= n (

p²

_
y

(1-p)²

(A.13)

To find the expected information we use the fact that the expected value of the sample mean [`y] is the population mean (1-p)/p, to obtain (after some simplification)

I(p) =

p²(1-p)

(A.14)

Note that the information increases with the sample size n and varies with p, increasing as p moves away from [2/3] towards 0 or 1.

In a sample of size n = 20, if the true value of the parameter was p = 0.15 the expected information would be I(0.15) = 1045.8. If the sample mean turned out to be [`y] = 3, the observed information would be 971.9. Of course, we don't know the true value of p. Substituting the mle [^(p)] = 0.25, we estimate the expected and observed information as 426.7.^[¯]

A.1.4 Newton-Raphson and Fisher Scoring

Calculation of the mle often requires iterative procedures. Consider expanding the score function evaluated at the mle [^(q)] around a trial value q₀ using a first order Taylor series, so that

^
q

) u(q₀) +

u(q)

(

^
q

-q₀).

(A.15)

Let H denote the Hessian or matrix of second derivatives of the log-likelihood function

H(q) =

²logL

u(q)

(A.16)

Setting the left-hand-size of Equation A.15 to zero and solving for [^(q)] gives the first-order approximation

^
q

= q₀ - H^-1(q₀) u(q₀).

(A.17)

This result provides the basis for an iterative approach for computing the mle known as the Newton-Raphson technique. Given a trial value, we use Equation A.17 to obtain an improved estimate and repeat the process until differences between successive estimates are sufficiently close to zero. (Or until the elements of the vector of first derivatives are sufficiently close to zero.) This procedure tends to converge quickly if the log-likelihood is well-behaved (close to quadratic) in a neighborhood of the maximum and if the starting value is reasonably close to the mle.

An alternative procedure first suggested by Fisher is to replace minus the Hessian by its expected value, the information matrix. The resulting procedure takes as our improved estimate

^
q

= q₀ + I^-1(q₀) u(q₀),

(A.18)

and is known as Fisher Scoring.

Example: Fisher Scoring in the Geometric Distribution. In this case setting the score to zero leads to an explicit solution for the mle and no iteration is needed. It is instructive, however, to try the procedure anyway. Using the results we have obtained for the score and information, the Fisher scoring procedure leads to the updating formula

^
p

= p₀ + (1-p₀-p₀

_
y

)p₀.

(A.19)

If the sample mean is [`y] = 3 and we start from p₀ = 0.1, say, the procedure converges to the mle [^(p)] = 0.25 in four iterations.^[¯]