Appendices

• The State of the Union Historically
• Flesch-Kincaid Scores Comparatively
• Statistical Methods Used in this Project
• Source Code

The State of the Union Historically

The following chart shows the timing of the messages and the distribution of oral and written delivery as well as indicating the increasing publicity of the address starting in the 20th Century.

Delivery: Oral in bold Written in normal weight *Radio after written marked with asterisk

Increase in publicity: No Broadcast Radio 1923 Television 1947 Evening 1965 Web 2002

(Source: https://www.presidency.ucsb.edu/documents/presidential-documents-archive-guidebook/annual-messages-congress-the-state-the-union)

Flesch-Kincaid Scores Comparatively

The Flesch-Kincaid score is commonly used for determining the age-appropriateness of reading material. It was developed originally in the 1940s by Rudolf Flesch, who wrote Why Johnny Can't Read. The current version of the formula was developed together with J.P. Kincaid for the Navy in the 1970s:

(0.39 * average_words_per_sentence)
+ (11.8 * average_syllables_per_word) - 15.59

It is a United States Government Department of Defense standard (DOD MIL-M-38784B). The score indicates that the text would be at the limit of comprehension for a person with the equivalent of that number of years of education; in a comprehension test, that person would answer 50 per cent of the questions correctly. It is estimated that the population of the U.S. has an average reading ability at the eighth-grade level.

The use of this metric is controversial. It does not account for sentence structure, vocabulary, style or context. This implementation is fraught with potential inaccuracies in the determination of the number of syllables in a word (possibly as much as ± 10%). It is probably best at comparing similar types of text. There are several newer, potentially more accurate measures of readability. The accuracy and applicability of the metric over 200 years is doubtful given changes in both language and education. So the use here of Flesch-Kincaid requires some explanation.

Flesch-Kincaid remains one of the best known and frequently used metrics of readability. Its correlation with grade-levels gives a simple and accessible sense of the metrics meaning. It is a convenient quantitative marker of style that has broad current acceptance. In examining the historical corpus of the State of the Union, a set of documents with exceptional continuity over time, it provides a measure of the gradual changes in language use.

One interesting finding of the project is in fact that the trend of the score is so consistent and seemingly uncorrelated with particular presidents. From this study it is not possible to determine whether this is a change in language generally or one specific to political language.

The following chart shows Flesch-Kincaid Scores of a variety of texts:

* Average Score

(Note: There is a rumor that USToday is written at 8th grade level, but I have found no documentation of that, and spot checks seem to indicate that it is probably in the same range—around 12—as other papers.)

(Sources: for Literature Amazon.com, for Newspapers Jack Hart, Editor & Publisher, November 6, 1993 (v126 i45 p.5) quoted at http://answers.google.com/answers/threadview?id=301734, for others original analysis.)

Statistical Methods Used in this Project

The methods used in this project all rely on frequency counts of words. The size of the word is determined simply by the frequency of its occurrence in the document. The x (horizontal) position within the interface is determined by the average position of the word in the document. The y position of displayed words (the vertical) is determined by a calculation of the words Relative Frequency Score, S.

The S score is an attempt to quantify significance as a combination of frequency and a determination of the uniqueness of the words use within an individual document as compared to its use in the corpus as a whole. S is average of two statistics: the Log Likelihood Statistic (LLS), the and the Term Frequency–Inverse Document Frequency (TF-IDF) . Each of these scores has been normalized so that the minimum and maximum values of each component over the corpus are comparable (between 0 and 100).

The formula for the LLS is:

LLS = 2f * ((freqInCorp * log(freqInCorp / E1)) + (freqInDoc * log(freqInDoc / E2)))

where

E1 = wordsInCorp * (freqInCorp + freqInDoc) / (wordsInCorp + wordsInDoc)
E2 = wordsInDoc * (freqInCorp + freqInDoc) / (wordsInCorp + wordsInDoc)

In order to even out the scale of the LLS, the following formula was employed to derive the log of the LLS+1, (L1LLS):

L1LLS = log(LLS+1)

The formula for TF:

TF = freqInDoc/wordsInDoc

The formula for IDF:

IDF = log( docsInCorpus / docsWhereWordOccurs)

The formula for TF-IDF:

TF-IDF = TF * IDF

The formula for S is as follows:

S = (L1LLS + TF-IDF) / 2
(values are normalized before being averaged)

Words are filtered to eliminate the most commonly used words.

For each document, the words with the top 40 S scores are selected. Depending on the length of the address, words with too few occurrences are also filtered out.

Data

• Word frequency data (normalized as frequency per 10,000 words) by year for all words in json file: history.json.br
• Analyzed word data for top words by document in JSON format (tiered by word count): 40 words | 60 words | 80 words
• Text of addresses: stateoftheunion1790-present.txt.br