Skip to content

Bridge “Academic Champ” Statistics Are Misleading (Dangerously So)

January 30, 2014

bridge academic champs and income I am often a fan of work that appears in Michigan’s online “The Bridge” but an article that appeared this week which provides “academic champ” ratings for school districts misses the mark completely and hits the wrong target.

Judging the quality of a school district’s work on the basis of test scores is simply not a useful measure since those scores depend heavily on factors outside of a school district’s control, particularly the socio-demographic characteristics of its students.  This article by the Bridge seems to takes important first step to solve the problem by attempting to control for socio-economic status (measured by the share of students receiving free and reduced school lunches).  What is left over after accounting for this difference should measure something like the “pure” contribution of the school.

But the measure used by Bridge (and calculated for it by Lansing-based Public Sector Consultants) simply does not do this.

The measure uses regression analysis to produce a baseline that theoretically factors out socio-economic status.  But the baseline they use simply does not do the job and what is left is /still/ heavily dependent on socio-economic status.  The strongest evidence of this is that the correlation between “academic champ” score and per capita income in Oakland County school districts is 0.85 (out of a possible 1.00), an incredibly strong relationship (I dream of finding such strong correlations in my own work).  bridge academic champs and income

(bridge academic champs in .pdf format)

The schools at the top of the “champ” ranking here are also the richest schools, the schools at the bottom are also the poorest and the line between them is almost perfectly straight.   (With the possible exception of South Lyons, no school district in Oakland County significantly over or under performs its raw per capita income).  An indicator cannot be said to control for income if every increase of $25,000 produces an increase of 6 points in the score (on a measure with a total range of only 50 points) and if this is true for almost every pairwise comparison in the dataset.  If a study controls for socio-economic status and still find an extremely strong relationship between performance and socio-economic status, then the study is not controlling enough and should not make any claims based on that control.

There is an analogy that might help make this clearer.  In the world of medical research there are lots of studies about the heredity involved in medical conditions.  Imagine a study about cholestol levels like the one done by Bridge that says “we should assume people have bad eating habits just because they have high cholesterol, since some of it is hereditary”, so we need to control for heredity.  Fortunately the researchers in this study have access to levels of maternal cholesterol and they do a new study that controls for that factor.  From this they note that that some people with high cholesterol are actually outperforming what their level /should/ be since their heredity predicted even higher levels, and they label these people as “cholesterol champs” and look to see what the champs did well as a source of recommendations for others.

All well and good, so far.  But the list also implicitly contains a list of “cholesterol duds”, and a lot of people look at the bottom of the list to confirm their previous impressions that some people are simply lazy and gluttonous.  And the study omits an important factor: that fathers may be an equally important factor in determining of cholesterol levels.  We need to get data on fathers’ levels, and can compare it to the “champ” numbers that control for mothers’ levels.  If, when we do, we find that there is no relationship between between the cholesterol levels of children and fathers, then we can be satisfied that the “champs” are doing something right (and maybe that the others are doing something wrong).  But if we find a very strong relationship–something like the .85 correlation found above), then we need to rethink the study because there are some “champs” who are clearly being helped by strong paternal genes rather than by any of their own choices, and there are some “duds” who face significant barriers to low cholesterol levels because they inherited a high propensity from their fathers that did not show up in the maternal study.

A study that looks at only some of the relevant control factors (overall income, parental education levels, per-pupil school allowances, percentage of /concentrated/ poverty), and (explicitly) rewards and (implicitly) punishes particular schools on the basis of those  incomplete controls undoes all of the positive outcomes that it intended to achieve.  Because the Bridge study fails to control for precisely what it claims to control, its “champ” (and “dud”) scores send exactly the wrong message.   It says that schools in rich areas are somehow doing a better job with their resources than those in poor areas even after control for their richness.   But the Bridge study simply fails to take into account /all/ of the resources that richer areas have, those that are not measured a regression based purely on school lunch figures.

Bridge needs to pull these calculations and rework them before presenting them back to us.  What is here is sadly misleading and, to the extent it influences how we think about who is “working hard” to educate our kids, it will lead us to the wrong conclusions about who is at fault and what we should do about it.

Advertisements
No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: