Problem Set 3: Refugee Appeals in Canada
Due Friday October 28, 2011
Greene and Shaffer (1992), cited in Fox (1997), analyzed decisions by the Canadian Federal Court of Appeals on cases filed by refugee applicants who had been turned down by the Immigration and Refugee Board. We are interested in the fact that there are large differences among judges.
It is possible, of course, that the judges get to hear very different cases. In our analysis we will control for an expert assessment of whether the case had merit, the city where the original application was filed (Toronto, Montreal or other) and the language in which it was filed (English, French). An additional predictor is the logit of the success rate for all cases from the applicant's country. The country itself is also available.
Fox's dataset is restricted to the 10 (of 12) judges who were present in
the court during the entire period, and to countries of origin that produced
at least 20 appeals during this period, and is available in the datasets
section of the course website as the Stata system file greene.dta.
There are 384 observations on 8 variables, most of which are strings.
[1] Differences by Judge
(a) Create a variable granted to indicate whether leave to appeal
was granted. This will be our outcome. (Decision is coded 'yes' or 'no'.)
(b) Tabulate the decisions by judge with row percents to verify that the proportion of cases where appeal is granted ranges from 10.3 for Iacobucci to 60% for Marceau.
(c) Generate dummy variables for the judges and fit a logistic regression model of granted on the judge dummies. Use Iacobucci as the reference judge to facilitate the comparisons that follow.
(d) Interpret the coefficient for judge Marceau. Verify that the predicted probability for this judge is 60%, in agreement with the crosstab.
(e) Test the hypothesis that the probability of granting leave to appeal is the same for all judges using (i) a likelihood ratio test, (ii) a Wald test.
[2] Adjusting for Merit, City, Language and Country
(a) Fit a model with the judge dummies and the following controls: (i) an indicator for whether the expert thought the case had merit, (ii) city indicators using 'Other City' as the reference cell, (iii) a language dummy with French as omitted category, and (iv) the logit of the country's success rate, as a way to control for country short of using 16 dummies.
(b) How would you justify the use of the logit of the country's success rate as a control, as opposed to country dummies? No need to fit any models, we want to keep this short. If you do, you'll have to figure out what to do with Fiji, India and Poland.
(c) Interpret briefly the coefficients of merit and of Montreal in this model in terms of odds ratios.
(d) Interpret the coefficient for judge Marceau in this model, explaining carefully what it means to adjust for the other predictors. Has the adjustment reduced the contrast with Iacobucci?
(e) Test the hypothesis that the probability of granting leave is the same for all judges after adjusting for the control variables using (i) a likelihood ratio test, and (ii) a Wald test.
[3] Adjusted Probabilities
(a) One way to present the results to a non-technical audience is to compute predicted probabilities for each judge setting all other variables in the model to their means. Do this calculation from first principles for Iacobucci and for Marceau and interpret the results.
(b) One problem with the previous approach is that setting a dummy to its mean is not very meaningful (0.36 Montreal?) Here's an alternative approach. Save your data. Set all judge dummies to zero, predict, and compute the mean probability. This is what the model predicts if Iacobucci ruled in all cases. Now set the Marceau dummy to one, predict and average the probabilities. This is what the model predicts if Marceau ruled in all cases. Compare results.
(c) Stata's powerful margin command can do these calculations
for you. Treat the judge as a factor variable (this requires integer codes),
and fit the model. Estimate the judge margins with the option
atmeans and compare with 3.a.
Estimate the judge margins with the default option and compare with 3.b.
Now you know what this command does.
[4] Goodness of Fit
(a) Test the goodness of fit of the model of part [2] and [3] using Stata's
default setting for the post-estimation command estat gof.
Can we trust the asymptotics given the number of distinct covariate patterns
relative to the number of observations?
(b) Compute the Hosmer-Lemeshow goodness of fit test using 10 groups based on deciles of predicted probabilities. Feel better now?
Posted Tuesday, October 18, 2011