MenuCollege ROI report drills down to individual majors at individual colleges? NOT SO FAST
BIG NEWS out about a new resource that examine college ROI in serious detail. You’ll almost certainly see news about it, so here’s a direct link to the main page at Third Way (the think tank that built it). In fact, it drills down to individual programs at every higher ed institution. So tempting to nerd out on the data . . .
So why is bunk?
Common Sense about Data
Determining a particular program’s ROI means that you HAVE TO HAVE two sets of numbers from the SAME SET OF STUDENTS (set aside for a second that for many ROI involves more than just money):
1) the prices they paid to get their degree . . . . the “I”
2) the salaries they received after graduation . . . the “R”
Although these folks have access to the “R”, they don’t have access to the “I”. And the proxy they use in place of having the actual “I” is the result of a boatload of unrelated data points that likely skews, if not outright corrupts, many of the results.
To accurately calculate an individual program’s ROI, the “R” has to comes from an average salary derived only from the students who graduated from that program. For example, seven sociology grads means seven salaries averaged together. So far so good; this data they have.
But the “I” they use isn’t taken from the same group of students at all. Instead, the “I” comes from ALL of that college’s graduates in the same year; hundreds or even thousands of them.
Given that the range of prices offered to different students at any college starts at zero and goes all the way up to the sticker price, what’s the chance the average price paid by those seven sociology majors is the same as the average price paid by all students at that school? HINT: not good.
It’s All About the Input
Assessing the ROI of a particular program at a particular college DEPENDS HEAVILY on the whether or not you can get the actual prices paid by the students who graduated from that program. Let’s say all of those sociology majors got jobs at non-profits that don’t pay much. If those students paid the average net price of a private college (about $33K) then the ROI for that program probably stinks.
But if most of those sociology majors entered that college as high merit award students and paid far less (let’s say $15K), then the ROI is almost certainly a big positive.
Without the actual prices that students pay, analyses like the ones in the article below are not helpful. They contribute to the narrative that all college is too expensive – a narrative that just isn’t based in fact.
Real Consequences
Moreover, there will undoubtedly be conversations at board of trustees meetings and in college president offices that will use these results to justify downsizing or eliminating programs. Worse still, which programs are the ones where these data findings are mostly likely to misstate the actual results? The ones with fewer majors per year. And which programs will those tend to be? In the humanities and fine arts.
I’m not arguing that college programs shouldn’t be evaluated, cut, downsized, revised, or re-envisioned. But I am arguing that, before colleges, policy makers, or the public starts down any other those paths, they ought to have accurate data.
Call me crazy.
