It looks like CNN credulously spit out another story from Global Language Monitor (GLM). Basically, GLM did their usual thing of running a speech (in this case, Obama’s oil spill speech from mid-June) through some mindless statistics, getting out the Flesch-Kincaid Grade Level, and then reporting it as though it was actually meaningful analysis. Language Log and Johnson already explained why the GLM analysis is nonsense, and as a result, CNN actually substantially re-wrote the story.

I discussed the meaninglessness of grade level analysis a year and a half ago in more depth, but this time let me just offer an illustration of why grade-level analysis is not at all appropriate for political analysis.  Here’s a bit from the early part of Obama’s address.  It has a Flesch-Kincaid Reading Level of 10.2, a level that GLM said reflected Obama’s “elite ethos”

“Already, this oil spill is the worst environmental disaster America has ever faced. And unlike an earthquake or a hurricane, it’s not a single event that does its damage in a matter of minutes or days. The millions of gallons of oil that have spilled into the Gulf of Mexico are more like an epidemic, one that we will be fighting for months and even years.”

Okay, but let me show you another passage that I’ve chosen to exactly match the above passage in Flesch-Kincaid Grade Level. It ought to be equally reflective of an elite ethos:

“one Gulf Already, America unlike millions even has oil does of spill ever be its minutes of not for disaster the And the single matter event earthquake we this epidemic, are a damage spilled The worst into environmental months it’s that or of a that of faced. will oil an is or like a hurricane, fighting Mexico more days. in an gallons that have and years.”

If the extent of your analysis is to look at grade levels, you’re going to say that these two passages are equivalent. That’s because the Flesch-Kincaid Grade Level formula is merely a weighted linear combination of number of words per sentence and number of letters per word. Since these two paragraphs contain the same words, letters, spaces, and periods, the statistics are the same for each, and therefore any conclusion drawn about the first paragraph solely from these statistics necessarily must be drawn about the second paragraph as well.

That’s the problem. These statistics and readability tests don’t look into word frequency, semantics, pragmatics, fluidity, rhetoric, style, or anything that actual humans do to assess the readability and meaningfulness of a text. The tests, after all, are intended as an approximation for when an informed analyst is not available, not as a data source in lieu of informed analysis.

To be fair, GLM’s analysis doesn’t stop at grade levels. They also offer the proportion of passive sentences in the address, which they report as “the highest level measured in any major presidential address this century”. And that’s something, except for Mark Liberman’s discovery that it’s not nearly true. Bush’s similar post-Katrina address had 17% passives; Obama’s post-oil-spill address lagged behind with a mere 11%. (GLM’s president, by the way, considers “There will be setbacks” to be a passive sentence, so it’s not terribly surprising that their passive statistics aren’t great.) But even if the count were right, the passive proportion is not an inherently meaningful statistic either, because passives are employed by good writers for reasons other than evasion, which seems to be the only use GLM can come up with for them.

I hesitate to say that there is no useful information to be found by calculating simple statistics on major presidential addresses. But readability scores are dependent on the choice of punctuation for a speech, overlook rhetorical devices and structure, ignore frequency and semantics, and haven’t been shown to correlate very well with listener comprehension. It is unlikely that useful information will come such simplistic analyses. And though it is not impossible that one day someone will find it, I have not yet seen a single informative result from grade level or other simple statistical analysis on political speech.