Sunday, October 08, 2006

Do Colleges "Lie" with Statistics?

I reviewed an article by Newsweek reporter and college/career advisor Marty Nemko’s “How Some Colleges Deceive Students, Parents”, which can be found at:
http://www.martynemko.com/pub/articles/college-deception.shtm.

Dr. Nemko is the author of “The All-in-One College Guide: a more-results, less-stress guide to choosing, getting into, finding the money for, and making the most of college.” He was dubbed, “The Ralph Nader of Education” by ABC-TV and holds a PhD specializing in evaluation of education from the University of California, Berkeley.

Dr. Nemko finds fault with several key statistics reported by colleges in the context of admissions recruiting. As I read the article, I changed my view on a key issue. I originally thought I was going to say that Nemko is fueling the fire of dissatisfaction. I maintain, though, that, for its flaws, we do have a college admissions process that strives to meet divergent, and admirable goals in ensuring equal access, while still maintaining a competitive environment that pushes colleges to excel; to ensure their survival and “thrival” (my word).

I had some misconceptions, though. I thought that most reported statistics in the FAQs distributed by school were consistent in their context. Additionally, I thought that most independent college guide books further ensure consistency by serving as the clearinghouse for such data.

I now realize that the FAQs certainly have the potential to be misleading and incomplete and that I need to pay better attention to any footnoting or “extra words” in the description of the statistic. Also guide books can certainly be outdated, and I don’t know how often this happens. However, on campus visits, I have been “corrected” on data reported in The Fiske Guide (or the admissions officer “spun” it another way for me, and I didn’t perceive the subtle difference). And I have read that sometimes colleges “miss” the opportunity to provide all the leading guides with all the updated needed in time for publication. (Would Dr. Nemko suggest this is intentional in a “down” year?)

Either way, bottom line, it’s a good idea to check the most critical facts, when using these data for final college selection, especially if they are important criteria in the decision.

Mr. Nemko dives deeper yet by accusing some colleges of actually “hiding” (read: misrepresenting) their total cost of attendance and graduation rates, since the average four-year graduation rate is lower than many students expect. (He quotes the four-year graduation rate at 37% at the time of his writing. Note the article is not dated.) It’s true that a fifth or sixth year can be a highly significant additional and unplanned cost, but this is largely within the control of the student. So I think Nemko’s furor is only partially justified on this point.

There are several other issues that Dr. Nemko has with reported statistics. However, for most of them, I feel he has failed to show intent to “lie with statistics”, as he puts it, since he has not shown examples where disclosures aren’t published along with the statistic. Naturally colleges are looking to present themselves in the best light, so the buyer must beware. However, when I go on campus visits, I think it’s amusing that statistics are quoted…or asked about….as if they mean anything to the individual student’s choice to attend. A statistic is only a directional indicator and, as an advisor, I am looking for all important indicators to be positive, when making a recommendation.

However, what’s important is different to different clients. The most important data points in the final analysis are invariably qualitative. I have never heard of a situation where a data point variance against another college has swayed a decision to attend or not to attend a specific school. So do colleges “lie with statistics”? I think they might “massage” the data to a most favorable point. But if we conclude that they “lie”, I say, so what?!

I do like Dr. Nemko’s suggestion to check on the diversity of a campus (assuming it’s important to some students) not by statistics, but by what kind of diversity classes are offered and what resources are available. And I like Dr. Nemko’s warning on guaranteed multi-year financial aid packages, i.e., to check if they will be adjusted for inflation or if the composition of the package will shift toward a higher percentage of loan versus grant over time. However, I have yet to meet a financial aid officer who has any authority to speculate or promise on these matters. So I highly doubt if a written commitment can be attained, as Dr. Nemko advises.