connector-sets.lyx

#LyX 2.0 created this file. For more info see http://www.lyx.org/
\lyxformat 413
\begin_document
\begin_header
\textclass article
\begin_preamble
\usepackage{url} 
\end_preamble
\use_default_options false
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding utf8
\fontencoding global
\font_roman times
\font_sans helvet
\font_typewriter courier
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100

\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks true
\pdf_pdfborder true
\pdf_colorlinks true
\pdf_backref false
\pdf_pdfusetitle true
\papersize default
\use_geometry false
\use_amsmath 2
\use_esint 0
\use_mhchem 0
\use_mathdots 1
\cite_engine basic
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\use_refstyle 0
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header

\begin_body

\begin_layout Title
Connector Sets
\end_layout

\begin_layout Author
Linas Vepstas
\end_layout

\begin_layout Abstract
Extract from the language-learning diary, reporting on the first small dataset
 containing connector sets.
 This is the 11 June 2017 update of the original 11 May 2017 report.
 It includes more data and figures, and updates the figures for readability
 (legibility).
 It also revises notation slightly.
\end_layout

\begin_layout Section*
Introduction 
\end_layout

\begin_layout Standard
This is a report on a small dataset of disjuncts and connector sets, extracted
 from MST parses of a batch of sentences.
 First, a recap of what these are, then a characterization of the database
 contents, and finally, a short report on the grammatical similarity of
 words in the dataset.
 
\end_layout

\begin_layout Subsection*
Summary of results
\end_layout

\begin_layout Standard
The primary results reported below are these:
\end_layout

\begin_layout Standard
* Most scores and metrics that can be assigned to connector sets give a
 (scale-free) Zipfian ranking distribution, and are thus fairly boring.
\end_layout

\begin_layout Standard
* The greater the average number of observations per disjunct, the more
 grammatically acceptable (accurate) the disjunct seems to be.
 This is good news: it means that the general technique is not generating
 ungrammatical garbage.
\end_layout

\begin_layout Standard
* Connector sets can be given a mutual information score.
 The ranking distribution for this is not at all Zipfian.
 The MI score seems to be quite good at identifying words that participate
 in idioms, set phrases and institutional phrases.
\end_layout

\begin_layout Standard
* The average number of connectors per disjunct, which should have indicated
 the part-of-speech that the word belongs to, fails to do this.
 This seems to be due to the small size of the dataset, the fact that it
 is polluted with lists and tables, masquerading as sentences (its a Wikipedia
 sample), and that there seem the be very few verbs in the sample (Wikipedia
 articles describe concepts and events, using the copula to describe them:
 
\begin_inset Quotes eld
\end_inset

is
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

has
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

was
\begin_inset Quotes erd
\end_inset

, and is almost devoid of narrative verbs: 
\begin_inset Quotes eld
\end_inset

ran
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

jumped
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

hit
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

ate
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

thought
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

took
\begin_inset Quotes erd
\end_inset

.) The WP sources need to be supplemented with narrative texts, ideally from
 young-adult literature.
\end_layout

\begin_layout Standard
* Cosine similarity applied to connector sets seems to be an effective way
 of determining the grammatical similarity of words.
 This is despite the fact that closer examination reveals that the dataset
 is quite thin and poor, and the similarity is based on scant evidence.
 Despite the scant evidence, the similarity scores do seem to distinguish
 different kinds of nouns quite well.
 A big surprise is that it also seems to cluster prepositions quite well.
 There seem to be few or no verbs in the dataset, as already noted.
 It seems that even small datasets are sufficient to get started with grammatica
l similarity clustering, but much larger samples are needed for discovering
 more sophisticated grammar.
\end_layout

\begin_layout Subsection*
Recap
\end_layout

\begin_layout Standard
The story so far: Starting from a large text corpus, the mutual information
 (MI) of word-pairs are counted.
 This MI is used to perform a maximum spanning-tree (MST) parse (of a different
 subset of) the corpus.
 From each parse, a pseudo-disjunct is extracted for each word.
 The pseudo-disjunct is like a real LG disjunct, except that each connector
 in the disjunct is the word at the far end of the link.
\end_layout

\begin_layout Standard
So, for example, in in idealized world, the MST parse of the sentence "Ben
 ate pizza" would produce the parse Ben <--> ate <--> pizza and from this,
 we can extract the pseudo-disjunct (Ben- pizza+) on the word "ate".
 Similarly, the sentence "Ben puked pizza" should produce the disjunct (Ben-
 pizza+) on the word "puked".
 Since these two disjuncts are the same, we can conclude that the two words
 "ate" and "puked" are very similar to each other.
 Considering all of the other disjuncts that arise in this example, we can
 conclude that these are the only two words that are similar.
\end_layout

\begin_layout Standard
Any given word will have many pseudo-disjuncts attached to it.
 Each disjunct has a count of the number of times it has been observed.
 Thus, this set of disjuncts can be imagined to be a vector in a high-dimensiona
l vector space, which each disjunct being a single basis element.
 The similarity of two words can be taken to be the cosine-similarity between
 the disjunct-vectors.
\end_layout

\begin_layout Standard
Equivalently, the set of disjuncts can be thought of as a weighted set:
 each disjunct has a weight, corresponding to the number of times it has
 been observed.
 A weighted set is more or less the same thing as a vector, and these two
 are treated as the same, in what follows.
 Note that the disjunct vectors are sparse: for any given word, almost all
 coefficients will have a count of zero.
 For example, the dataset that will be examined next has over a quarter
 of a million different pseudo-disjuncts in it; most words have fewer than
 a hundred disjuncts on them.
\end_layout

\begin_layout Standard
Some terminology and notation are introduced next, followed by a characterizatio
n of the dataset.
 This is followed by a statistical analysis of the word-disjunct pairs,
 and is followed by an analysis of the resulting word-similarity.
\end_layout

\begin_layout Subsection*
Terminology
\end_layout

\begin_layout Standard
It is useful to introduce some notation for counting words, disjuncts, and
 connectors.
 Let 
\begin_inset Formula $N(w)$
\end_inset

 be the number of times that the word 
\begin_inset Formula $w$
\end_inset

 has been observed, in the dataset.
 Let 
\begin_inset Formula $N(w,d)$
\end_inset

 be the number of times that the disjunct 
\begin_inset Formula $d$
\end_inset

 has been observed on word 
\begin_inset Formula $w$
\end_inset

.
 The pair 
\begin_inset Formula $(w,d)$
\end_inset

 is referred to as a 
\begin_inset Quotes eld
\end_inset

connector set
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

cset
\begin_inset Quotes erd
\end_inset

 in the text below.
 Thus, for a word 
\begin_inset Formula $w$
\end_inset

, there is a set 
\begin_inset Formula $(w,*)=\left\{ (w,d)|N(w,d)>0\right\} $
\end_inset

 of associated csets, called the 
\begin_inset Quotes eld
\end_inset

support
\begin_inset Quotes erd
\end_inset

 of the word.
 The size of this set can be written using the standard notation for set-sizes
 as 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

.
 Similarly, a disjunct 
\begin_inset Formula $d$
\end_inset

, is supported by the set 
\begin_inset Formula $(*,d)=\left\{ (w,d)|N(w,d)>0\right\} $
\end_inset

 of associated csets.
 
\end_layout

\begin_layout Standard
The primary contents of the database are the counts 
\begin_inset Formula $N(w,d)$
\end_inset

 and everything else of interest in this section can be obtained from this.
 Note that 
\begin_inset Formula $N(w,d)$
\end_inset

 can be understood as a matrix, where the disjuncts identify columns, and
 the words identify rows.
 In general, this is a very sparse matrix: the number of non-zero entries
 
\begin_inset Formula $\left|(*,*)\right|$
\end_inset

 is far less than the number of rows times the number of columns.
\end_layout

\begin_layout Standard
Every time a word is observed in an MST parse, a disjunct is extracted for
 it; thus, word observations and disjunct observations are on one-to-one
 correspondence.
 In notation:
\begin_inset Formula 
\[
\sum_{d}N(w,d)=N(w,*)=N(w)
\]

\end_inset

Similarly, the total number of times that a disjunct was observed is just
\begin_inset Formula 
\[
N(*,d)=\sum_{w}N(w,d)
\]

\end_inset


\end_layout

\begin_layout Standard
Frequencies can be obtained by dividing by the total number of observations,
 so that 
\begin_inset Formula $p(w,d)=N(w,d)/N(*)$
\end_inset

 and 
\begin_inset Formula $p(w)=N(w)/N(*)$
\end_inset

 with 
\begin_inset Formula $N(*)=\sum_{w}N(w)$
\end_inset

 the total number of observations of words.
\end_layout

\begin_layout Standard
A single disjunct is always composed of a fixed number of connectors, independen
tly of any observations; let 
\begin_inset Formula $C(d,c)$
\end_inset

 be the number of times that connector 
\begin_inset Formula $c$
\end_inset

 appears in disjunct 
\begin_inset Formula $d$
\end_inset

.
 Note that 
\begin_inset Formula $C(d,c)$
\end_inset

 is almost always either zero or one; however, a connector can appear more
 than once in a disjunct, so this count can rise to 2 or 3 or very rarely
 higher.
 The wild-card sum 
\begin_inset Formula $C(d,*)=\sum_{c}C(d,c)$
\end_inset

 is the total number of connectors in the disjunct; it is the vertex degree
 of all edges connecting to that disjunct.
 It is also useful to define 
\begin_inset Formula $C(d,+)$
\end_inset

 and 
\begin_inset Formula $C(d,-)$
\end_inset

 as the total number of right-linking and left-linking connectors.
\end_layout

\begin_layout Subsection*
Dataset characterization
\end_layout

\begin_layout Standard
The following charts and analyses are derived from a single dataset, a relativel
y small dataset, taken as a snapshot during counting.
 Its called 'en_pairs_sim'.
 It contains 37413 words that have disjuncts attached to them.
 These words have been observed a total of 661104 times, for an average
 of 661104/37413 = 17.6 observations per word.
 This dataset contains 291637 different, unique disjuncts, for an average
 of 661104/291637 = 2.27 observations per disjunct.
 The period appears 32536 times, suggesting that this many sentences were
 observed.
 Each sentence thus has an average of 661104/32536 = 20.3 words per sentence.
 The dataset contains 446204 unique connector-sets, for an average of 661104/446
204 = 1.48 observations per cset.
 It is this last number that makes this dataset feel thin and sparse.
\end_layout

\begin_layout Standard
The dataset is sparse in a completely different sense: viewing 
\begin_inset Formula $N(w,d)$
\end_inset

 as a matrix whose size is 
\begin_inset Formula $37413\times291637$
\end_inset

, but only a very small number of these is non-zero: this is 
\begin_inset Formula $446204/(37413\times291637)=4.089\times10^{-5}$
\end_inset

.
 The sparsity of this matrix can be defined as 
\begin_inset Formula $-\log_{2}$
\end_inset

 of this number, which is 14.58.
\end_layout

\begin_layout Standard
The total word-entropy for the dataset is defined as
\begin_inset Formula 
\[
H_{word}=-\sum_{w}p(w)\log_{2}p(w)
\]

\end_inset

and was measured to be 
\begin_inset Formula $H_{word}=10.28$
\end_inset

 bits.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
This and the following entropies were measured with the word-entropy-bits,
 disjunct-entropy-bits, etc.
 functions in disjunct-stats.scm 
\end_layout

\end_inset

 The connector-set entropy is much higher.
 It is defined as
\begin_inset Formula 
\[
H_{cset}=-\sum_{w,d}p(w,d)\log_{2}p(w,d)
\]

\end_inset

and is measured to be 
\begin_inset Formula $H_{cset}=18.30$
\end_inset

 bits.
 The disjunct entropy is dual to the word entropy:
\begin_inset Formula 
\[
H_{disjunct}=-\sum_{d}p(*,d)\log_{2}p(*,d)
\]

\end_inset

and is measured to be 
\begin_inset Formula $H_{disjunct}=16.01$
\end_inset

 bits.
 The total mutual information between the words and disjuncts is then 
\begin_inset Formula 
\[
MI_{cset}=-\sum_{w,d}p(w,d)\log_{2}\frac{p(w,d)}{p(*,d)p(w,*)}=H_{cset}-H_{word}-H_{disjunct}
\]

\end_inset

and is measured to be 
\begin_inset Formula $MI_{cset}=-7.985$
\end_inset

 bits.
\end_layout

\begin_layout Subsection*
Connector-set distribution
\end_layout

\begin_layout Standard
Some connector-sets will be observed far more often than others.
 Likewise for the two sides of the connector-set: some words will have far
 more observations, and some disjuncts will be seen more often.
\end_layout

\begin_layout Standard
Two graphs, dual to one-another.
 The one on the left shows 
\begin_inset Formula $N(w,*)$
\end_inset

, ranked by count.
 The one on the right shows 
\begin_inset Formula $N(*,d)$
\end_inset

, also ranked.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Obtained by running (print-ts-rank sorted-word-obs outport) from the disjunct-st
ats.scm file, on the en_pairs_sim database.
 The second one prints sorted-dj-obs.
\end_layout

\end_inset

.
 The first follows the canonical Zipf distribution.
 The green line is an eyeballed, approximate fit, of exponent -1.1.
 The second has an exponent of -0.85.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-word-obs.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename ../images/ranked-dj-obs.eps
	width 50text%

\end_inset


\end_layout

\begin_layout Standard
The first ten words in the word ranking are: "the" "," "." "of" "and" "in"
 "to" "a" "was".
 This is the ranking of how often these words appear, overall, in the MST-parsed
 corpus.
 The number of periods should be equal to the number of sentences in the
 corpus; commas and the word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 can appear more than once in a sentence.
 This list is repeated in the table below.
 The frequency is just the count divided by 661104.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="10" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
frequency
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $-\log_{2}$
\end_inset

frequency
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
38977
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0589
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.084
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
,
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
37524
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0568
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.139
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32536
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0489
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.353
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
of
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
22654
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0343
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.867
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17708
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0268
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.222
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14900
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0225
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.471
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12825
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0194
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.688
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11882
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0180
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.798
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
was
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6970
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0105
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6.568
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
The main point of this table is to demonstrate the log-likelihood column.
 At this point, these numbers won't seem to have much meaning; however,
 they provide an overall scale that will be seen, repeatedly, in the analysis
 below.
 The range of magnitudes -- 4 to 7 -- is no accident, and similar ranges
 will be seen later.
 
\end_layout

\begin_layout Standard
The first ten pseudo-disjuncts in the disjunct-ranking are ".+ " "the- "
 "the+ " ",+ " ",- " "of+ " "of- " "a- " "and+ " "and- ".
 The meaning of the plus and minus signs was explained above; but to recap:
 the disjunct 
\begin_inset Quotes eld
\end_inset

.+
\begin_inset Quotes erd
\end_inset

 means that there are many words that expect to be followed by a period
 (on the right).
 The disjunct 
\begin_inset Quotes eld
\end_inset

the-
\begin_inset Quotes erd
\end_inset

 means that there are many words that want to link to the word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 on the left.
 This is grammatically correct: 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 is a determiner, and it is always the dependent of some noun.
 The disjunct 
\begin_inset Quotes eld
\end_inset

the+
\begin_inset Quotes erd
\end_inset

 is grammatical garbage/nonsense: it states that there are many words that
 want to link to the word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 on the right.
 This is never correct for English; determiners always precede the noun
 that they modify.
 The rest of the disjunct look reasonable: it's legitimate to link to commas
 on both the left and right, and likewise to the preposition 
\begin_inset Quotes eld
\end_inset

of
\begin_inset Quotes erd
\end_inset

 and conjunction 
\begin_inset Quotes eld
\end_inset

and
\begin_inset Quotes erd
\end_inset

.
 The disjunct 
\begin_inset Quotes eld
\end_inset

a-
\begin_inset Quotes erd
\end_inset

 is correct, and the insane disjunct 
\begin_inset Quotes eld
\end_inset

a+
\begin_inset Quotes erd
\end_inset

 doesn't appear until the 14th slot.
\end_layout

\begin_layout Standard
It's not clear at this point why the insane disjuncts 
\begin_inset Quotes eld
\end_inset

the+
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

a+
\begin_inset Quotes erd
\end_inset

 are so highly ranked.
 The hope is that these appear due to the poor quality of the dataset (as
 will become more clear, below), rather than a deficiency in the theory.
 Based on decades-old research on MST parsing, its reasonable to hope that
 these disjuncts will disappear from a larger, better corpus.
\end_layout

\begin_layout Standard
The ranking of connector sets is shown below.
 It's a graph of the ranked counts 
\begin_inset Formula $N(w,d)$
\end_inset

.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-cset-obs.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
The top-ten connector-sets are ".: )-" "and: ,-" "in: the+" "(: )+" ",: and+"
 "the: of-" "of: the+" "): .+" "He: was+".
 These are hard to read, so, decoded: the first is a parenthesis followed
 by a period.
 The second is a comma, followed by 
\begin_inset Quotes eld
\end_inset

and
\begin_inset Quotes erd
\end_inset

.
 The third is the word-pair 
\begin_inset Quotes eld
\end_inset

in the
\begin_inset Quotes erd
\end_inset

.
 All the others are pairs, also.
 Written in order: 
\begin_inset Quotes eld
\end_inset

()
\begin_inset Quotes erd
\end_inset

 (presumably with other text between the parens) 
\begin_inset Quotes eld
\end_inset

, and
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

of the
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

of the
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

) .
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

He was
\begin_inset Quotes erd
\end_inset

.
 Curiously the 1st and 8th give the same pair, as do 2nd and 5th, and the
 6th and 7th.
 This makes sense: if one word-disjunct pair is observed a lot, the converse
 should be also.
 This duality might be behind the slope of slightly less than -0.6 in the
 graph.
\end_layout

\begin_layout Standard
It is also interesting to turn this graph 
\begin_inset Quotes eld
\end_inset

on it's side
\begin_inset Quotes erd
\end_inset

.
 So, in this dataset, there were 15270 words observed exactly once (out
 of a total of 661104 observations of 37413 words).
 This is quite something: of all the words observed, almost half were seen
 only once.
 More than half were seen twice, or less.
 These are presumably rare typos, foreign words, IPA pronunciation guides:
 any word that appears only once must be unusual; and yet, there are a lot
 of them! There are 5790 words that appear twice, 3093 that appear exactly
 3 times, 
\emph on
etc.

\emph default
 These counts are graphed below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
graph of binned-word-counts.dat, generated in disjunct-stats.scm 
\end_layout

\end_inset

 
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/binned-word-counts.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
This graph indicates that most (almost all) words were observed less than
 100 times.
 In this dataset, there were only 630 words that were observed more than
 100 times, and 267 words that were observed 200 or more times.
\end_layout

\begin_layout Standard
Writing 
\begin_inset Formula $N$
\end_inset

 for the number of times that some word was observed, it appears that there
 are approximately 
\begin_inset Formula $15270\times N^{-3/2}$
\end_inset

 words observed that many times.
 In formulas, the size of the set of words 
\begin_inset Formula $\left\{ w|N(w)=N\right\} $
\end_inset

is given by 
\begin_inset Formula 
\[
\left|\left\{ w|N(w)=N\right\} \right|\sim N^{-3/2}
\]

\end_inset

where 
\begin_inset Formula $\left\{ w|\mbox{cond}\right\} $
\end_inset

 is a set of words (subject to the condition cond) and 
\begin_inset Formula $\left|\left\{ w|\mbox{cond}\right\} \right|$
\end_inset

 denotes the size of that set of words.
\end_layout

\begin_layout Standard
The next graph belabors the point, and yet it's important.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Generated from binned-word-logli.dat
\end_layout

\end_inset

 It shows nothing new, but it does show it in a format that will be recur
 frequently, later.
 Thus, its worth understanding now.
 This graph shows exactly the same data as the previous graph: it 
\emph on
is
\emph default
 the same graph, except that the x-axis is now labeled differently, and
 some of the counts have been binned together.
 So first: note that 
\begin_inset Formula $-\log_{2}1/661104=19.334$
\end_inset

 and so this is the location of the first spike on the far-right.
 Next, 
\begin_inset Formula $-\log_{2}2/661104=18.334$
\end_inset

 and 
\begin_inset Formula $-\log_{2}3/661104=17.750$
\end_inset

 are the locations of the second and third spikes: these correspond to words
 that have been observed 1,2 and 3 times.
 The height of the spikes are 
\begin_inset Formula 
\[
\left|\left\{ w|N(w)=N\right\} \right|\sim2^{-3/2\times\log_{2}N}
\]

\end_inset

 as clearly demonstrated by the straight green line.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/binned-word-logli.eps
	width 70text%

\end_inset


\begin_inset CommandInset label
LatexCommand label
name "logli distribution graph"

\end_inset


\end_layout

\begin_layout Standard
One peculiar difference between this, and the previous graph is the use
 of binning.
 Notice here that the observed, red line bounces 
\emph on
above
\emph default
 the straight green line, while in the previous graph, it was strictly below.
 If these both show the same data, how can this be? This seems to be a bit
 of a paradox.
 The difference can be understood as follows (and is important to keep in
 mind).
 This graph was generated by bin-counting.
 The x-axis was divided into 200 equal-sized bins, and whenever the log-likeliho
od of a word landed within a particular bin, the count was accumulated into
 that bin.
 Both this and the previous graph are distributions, but they use different
 measures.
 The two measures are related by the Jacobian determinant
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant
\end_layout

\end_inset

, which is given by the derivative of the log.
 This is a curious effect, to be kept in mind when comparing different graphs.
\end_layout

\begin_layout Subsection*
Ranked average observations per disjunct
\end_layout

\begin_layout Standard
A more interesting distribution arises by looking at the average number
 of observations per disjunct (per word).
 That is, a single word may have hundreds of disjuncts, observed thousands
 of times; what is the average number of times that a disjunct is observed?
 By 
\begin_inset Quotes eld
\end_inset

average
\begin_inset Quotes erd
\end_inset

, it is explicitly meant 
\begin_inset Formula $N(w,*)/\left|(w,*)\right|$
\end_inset

, the number of observations divided by the support for those observations.
\end_layout

\begin_layout Standard
This number gives a hint of how 
\begin_inset Quotes eld
\end_inset

narrow
\begin_inset Quotes erd
\end_inset

 the grammatical usage of a word is.
 If the average is high, it means that the word just does not have very
 many disjuncts on it; the few that it does have are observed a lot.
 Recall that these disjuncts (pseudo-disjuncts) connect to individual words,
 and not to word-classes.
 Thus, if a disjunct is seen a lot, it probably connects to another word,
 forming a high-MI pair.
 This can be explicitly seen in the example further below.
\end_layout

\begin_layout Standard
A graph of the ranked average number of observations, per disjunct, per
 word, is shown on the left, below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with the sorted-avg list in disjunct-stats.scm
\end_layout

\end_inset

 Words with fewer than 100 observations have been excluded, so as to minimize
 the spurious noise.
 Note that although the line is straight in this log-log graph, the Zipf
 exponent is approximately -1/4.
 The graph on the right expresses an alternate view of the same idea: this
 one shows a bin-count of all of the words.
 Reaffirming the graph on the left, it indicates that almost all words have
 an average disjunct observation count of less than three.
 Both of these graphs emphasize just how thin and weak this dataset is:
 to be certain of a disjunct, it sure would be nice to observe it, in actual
 use, more than half-a-dozen times! 
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-avg.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename ../images/binned-avg.eps
	width 50text%

\end_inset


\end_layout

\begin_layout Standard
The spike on the far right on the graph on the right suggests that there
 are about 30 words that have an average number of observations of greater
 than 6! Since they are 
\emph on
not
\emph default
 showing up in the graph on the left, this shows that the underlying word
 have fewer than 100 observations.
\end_layout

\begin_layout Standard
The first ten on the ranked list are "According" "Gestalt" "Cao" "Berger"
 "2015" "It" "U.S" "United" "However" "y".
 Unsurprisingly, most of these are all capitalized, suggesting that they
 are either part of a proper name, or that perhaps they begin a sentence.
 
\end_layout

\begin_layout Standard
For example, the appearance of 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

 is almost surely due to the high-MI pair 
\begin_inset Quotes eld
\end_inset

United States
\begin_inset Quotes erd
\end_inset

, and so this probably dominates the disjunct count.
 This is confirmed by manual inspection
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
View disjuncts by saying (filter (lambda (cset) (< 10 (get-count cset)))
 (cog-incoming-by-type (Word "foo") 'LgWordCset)) where 10 is the minimum
 number of counts.
\end_layout

\end_inset

, per table below.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
number of observations
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- States+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
108
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in- the- States+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
41
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
States+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
35
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- Kingdom+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- the- States+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Only the first four disjuncts seem to be grammatically acceptable.
 In the first, second and fourth usage, 
\begin_inset Quotes eld
\end_inset

States
\begin_inset Quotes erd
\end_inset

 seems to be a head word, with 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

 being the dependent.
 In the third case, it would appears that 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

 is the head-word, and 
\begin_inset Quotes eld
\end_inset

States
\begin_inset Quotes erd
\end_inset

 the dependent.
 
\end_layout

\begin_layout Standard
The fifth most frequent disjunct is grammatical garbage: noise from bad
 MST parses.
 There are even more disjuncts, which occur with lower frequency: there
 are 156 in all.
 Almost all of these are presumably junk of some kind.
 We conclude that the signal/noise ratio here is about 
\begin_inset Formula $10\log_{10}(108/14)$
\end_inset

 or about 9 dB.
 These first four disjuncts really pull up the average, which is quite low,
 given this skew: the average number of observations per disjunct for 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

 is 2.96.
\end_layout

\begin_layout Standard
The corresponding table for 
\begin_inset Quotes eld
\end_inset

It
\begin_inset Quotes erd
\end_inset

 is
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
number of observations
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
is+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
216
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
was+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
197
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
has+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
55
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
was+ was+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
is+ was+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
This also is much as expected: 
\begin_inset Quotes eld
\end_inset

It
\begin_inset Quotes erd
\end_inset

 is a sentence opener, and the three sentence types it opens are 
\begin_inset Quotes eld
\end_inset

It is
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

It was
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

It has
\begin_inset Quotes erd
\end_inset

.
 The fourth and fifth disjuncts are grammatical garbage.
 The average number of disjunct observations is 3.03, almost identical to
 that for 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Standard
The skewness appears to be very sharp.
 This suggests that we should not waste time looking at mean-square variations
 in the average, although we'll do this anyway.
 But first, its worth graphing the skewness directly.
 Again, this is done on a log-log graph, in a Zipfian way.
\end_layout

\begin_layout Subsection*
Disjunct count distribution
\end_layout

\begin_layout Standard
The graph below shows the distribution of the disjunct observations on the
 two words 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

It
\begin_inset Quotes erd
\end_inset

.
 Indeed, it looks Zipfian; since we know that all but the first three or
 four disjuncts are noise, this graph illustrates 
\begin_inset Quotes eld
\end_inset

pink noise
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

1/f noise
\begin_inset Quotes erd
\end_inset

.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with the dj-united and dj-it arrays indisjunct-stats.scm 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-dj-united.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Can one get a smoother distribution by summing together these two graphs?
 Sure...
 and one can sum together not just these two words, but all words (that
 have been observed at least 100 times).
 That graph is shown below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with the accum-dj-all function in disjunct-stats.scm 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-dj-counts.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Its kind of a strange graph.
 Yes, the x-axis of this graph does imply that there are dozens, if not
 hundreds of words with more than a thousand disjuncts on them, and maybe
 half-a-dozen that have more than ten-thousand (unique, different) disjuncts
 on them! Exactly what does this mean? This is covered in the next section.
\end_layout

\begin_layout Subsection*
Disjunct Support Distribution
\end_layout

\begin_layout Standard
Is it possible that some words have a large number of disjuncts on them?
 Yes, it is.
 For example, the period was observed to have 18200 unique, different disjuncts
 associated with it.
 The word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 has 15420 unique, different disjuncts, the comma has 14510.
 Rounding out this list are "of" "and" "in" "a" "to" "was".
 Its not clear what fraction of these disjuncts are grammatically valid,
 and what fraction are junk.
\end_layout

\begin_layout Standard
The graph below shows the distribution of the size of the support: the ranking
 of 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

.
 Again, the graph appears to be approximately Zipfian.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Generated by sorted-support in disjunct-stats.scm 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-support.eps
	width 60text%

\end_inset


\end_layout

\begin_layout Standard
Terminology: the 
\begin_inset Quotes eld
\end_inset

support
\begin_inset Quotes erd
\end_inset

 of a vector is the number of basis elements that have a non-zero coefficient.
 This is the set 
\begin_inset Formula $(w,*)$
\end_inset

 defined earlier.
 Equivalently, this is the size of the set of disjuncts associated with
 a word, when counted 
\emph on
without
\emph default
 multiplicity.
\end_layout

\begin_layout Subsection*
Ranked Euclidean length (RMS Size)
\end_layout

\begin_layout Standard
A different distribution arises by looking at the ranked RMS sizes of the
 disjunct sets
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Obtained by running (print-ts-rank sorted-lengths outport) from the disjunct-sta
ts.scm file on the en_pairs_sim database.
 
\end_layout

\end_inset

.
 Here, the RMS size
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
The word 
\begin_inset Quotes eld
\end_inset

length
\begin_inset Quotes erd
\end_inset

 can be used to describe the root-mean-square size of the set of disjuncts
 associated with a word.
 That is, each element of the set is a disjunct, and that disjunct has a
 count, the number of times it has been observed.
 The root-mean-square of these counts can be taken as the set-size.
 But this set can also be interpreted as a vector, and so the RMS size is
 the same thing as the Euclidean length of the vector.
 Thus, the word 
\begin_inset Quotes eld
\end_inset

length
\begin_inset Quotes erd
\end_inset

 is sometimes used for the RMS size; they're the same thing.
\end_layout

\end_inset

 is computed by taking the root-mean-square of the counts on each disjunct
 in the set, that is, by computing 
\begin_inset Formula $\sqrt{\sum_{d}N(w,d)^{2}}$
\end_inset

 for each word 
\begin_inset Formula $w$
\end_inset

 and then ranking.
 Interpreting 
\begin_inset Formula $d$
\end_inset

 as a basis element of a vector space, this can be recognized as the Euclidean
 length of the count-vector.
\end_layout

\begin_layout Standard
The RMS size of the set is thus larger not only when more disjuncts have
 been observed, but also when most of the observations are made of only
 a small handful of disjuncts.
 That is, the RMS size should be relatively larger, if the word is less
 grammatically flexible.
 So for example, prepositions tend to be very flexible; adjectives, not
 so much.
 Thus, we expect adjectives to appear higher-up on this ranking list, than
 the observation-based list.
 And this might be true, relatively, but certainly not true absolutely.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-lengths.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
The first ten words of the RMS-size-list are: "." "and" "," "in" "the" ")"
 "(" "of" "as" "to".
 Not that interesting: these are all words that were observed a lot in the
 text.
 The RMS size is dominated by the total number of observations of a word
 in text.
 In and of itself, its insufficient to indicate how 
\begin_inset Quotes eld
\end_inset

concentrated
\begin_inset Quotes erd
\end_inset

 the disjuncts are, how grammatically narrow a word is.
 For this, some other quantity is needed.
\end_layout

\begin_layout Standard
Note that the graph above is Zipfian, but the slope is approximately -0.65.
 The green line is an 
\begin_inset Quotes eld
\end_inset

eyeball
\begin_inset Quotes erd
\end_inset

 fit.
 Why the slope is approximately 2/3rds of the canonical slope is not clear.
\end_layout

\begin_layout Subsection*
Mean-square to size ratio
\end_layout

\begin_layout Standard
More interesting is the ratio of mean-square size to the total size.
 In formulas, by ranking according to 
\begin_inset Formula 
\[
\frac{\sqrt{\sum_{d}N^{2}(w,d)}}{N(w,*)}=\frac{\sqrt{\sum_{d}p^{2}(w,d)}}{p(w,*)}
\]

\end_inset

This seems like the interesting ratio, because the Zipf exponent of -0.65
 would be doubled, when working with mean-square sizes, thus making the
 two rankings comparable.
\end_layout

\begin_layout Standard
Words high in this score will be words that have relatively few disjuncts
 on them, or at least, few that matter much, that rise above the level of
 noise.
 The first ten on this list are (after excluding all words with fewer than
 100 observations): "It" "According" "He" ")" "and" "(" "United" "as" "."
 "well".
 Note that most are capitalized: that means that these words appeared at
 the beginning of sentences; the way in which the can link to what follows
 is fairly constrained, and thus it is no surprise that these have only
 a few disjuncts on them.
 
\end_layout

\begin_layout Standard
Excluding the capitalized words, and punctuation, in this list what remains
 is quite surprising.
 It is shown in the table below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Extracted from ranked-sqlen-norm.dat 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
score
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5 
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
41.4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31.6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
as
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30.1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
well
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
29.9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18.8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
first
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17.2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
is
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
22
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16.7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
also
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15.3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
can
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15.3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
on
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14.9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
at
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset

 
\begin_inset space \qquad{}
\end_inset

  
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
score
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14.4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
been
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
13.2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
but
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
13.1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
part
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
34
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12.8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
with
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
35
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12.8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
old
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
36
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12.7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
same
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
38
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11.9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
one
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
39
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11.6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
which
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
43
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
not
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
49
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset

 
\end_layout

\begin_layout Standard
It is surprising because, grammatically, we expect most of these words to
 have a large number of varied disjuncts attached to them.
 We expect them to be diffuse, not sharp: we expect that these would have
 a large number of observations smeared over a large variety of disjuncts,
 instead of having their weight concentrated in only one or two disjuncts,
 the way that 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

 was, above.
 So what is going on, here?
\end_layout

\begin_layout Standard
The distribution is unusual; it is shown below.
 Note that the x-axis is drawn with a log-scale.
 It appears to be linear, with an exponent of -4.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/binned-sqlen-norm.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Are there other interesting measures? One could contemplate the ratio of
 the mean-square size to the support 
\begin_inset Formula $\sum_{d}N(w,d)^{2}/\left|(w,*)\right|$
\end_inset

.
 Another possibility would be this, minus the average-squared, which would
 give the second moment, aka, the mean-square deviation from the average,
 specifically
\begin_inset Formula 
\[
\frac{\sum_{d}N(w,d)^{2}}{\left|(w,*)\right|}-\left[\frac{N(w,*)}{\left|(w,*)\right|}\right]^{2}=\frac{1}{\left|(w,*)\right|}\sum_{d}\left[N(w,d)-\frac{N(w,*)}{\left|(w,*)\right|}\right]^{2}
\]

\end_inset

Neither of these variations seem promising; they seem to offer up more of
 the same, at least on this dataset.
 A larger and more refined dataset might reveal otherwise.
\end_layout

\begin_layout Subsection*
Mutual information
\end_layout

\begin_layout Standard
The concept of the 
\begin_inset Quotes eld
\end_inset

fractional mutual information
\begin_inset Quotes erd
\end_inset

 for a pair is interesting to explore.
 Define this as
\begin_inset Formula 
\begin{equation}
MI_{pair}(w,d)=-\log_{2}\frac{p(w,d)}{p(w,*)p(*,d)}\label{eq:mi-word-disjunct}
\end{equation}

\end_inset

This is 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 in the sense that the total MI for the set of all pairs can be written
 as
\begin_inset Formula 
\[
MI_{cset}=\sum_{w,d}p(w,d)MI_{pair}(w,d)
\]

\end_inset

Fractional MI is interesting because it usually has a reasonably nice distributi
on.
 For this particular dataset, it ranges from about -20 to +8.
 The distribution is shown in the graphs below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
These are graphed by binned-cset-mi and weighted-cset-mi in the disjunct-stats.sc
m file.
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/binned-cset-mi.eps
	width 50col%

\end_inset


\begin_inset Graphics
	filename ../images/weighted-cset-mi.eps
	width 50col%

\end_inset


\end_layout

\begin_layout Standard
These graphs are generated by computing the value for 
\begin_inset Formula $MI_{pair}(w,d)$
\end_inset

 for each of the 446204 
\begin_inset Formula $(w,d)$
\end_inset

 pairs (aka 'connector sets'), and approximating it's distribution by bin
 counting.
 In each graph above, there are 200 bins, each of width of about 30/200,
 and each pair is assigned to one of the binds, according to it's MI value.
 The graph on the left then shows how many pairs there are in each bin.
 The graph on the left is similar, but not the same: it sums the frequencies
 for all the pairs in each bin.
 In formulas: the graph on the left shows the value of
\begin_inset Formula 
\[
\mbox{sizeof }\left\{ (w,d)|MI_{pair}(w,d)\mbox{ is in bin}\right\} 
\]

\end_inset

while the graph on the right shows
\begin_inset Formula 
\[
\sum_{MI_{pair}(w,d)\mbox{ is in bin}}p(w,d)
\]

\end_inset

where '
\begin_inset Formula $x\mbox{ is in bin}$
\end_inset

' simply means 
\begin_inset Formula $lo\le x<hi$
\end_inset

 with the bin being the interval 
\begin_inset Formula $[lo,hi)$
\end_inset

.
\end_layout

\begin_layout Standard
Both of these graphs show 
\begin_inset Quotes eld
\end_inset

combs
\begin_inset Quotes erd
\end_inset

 in the left side.
 These combs are exactly the same combs as noted in the last figure in section
 
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"

\end_inset

 
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"

\end_inset

.
 The combs are due to the large number of words that have been observed
 only a small handful of times.
 In essence, the combs attest that the bulk of the high-MI pairs have been
 observed more than just a few times; 
\emph on
i.e.

\emph default
 the high MI values are meaningful.
\end_layout

\begin_layout Subsection*
Mutual Information of words
\end_layout

\begin_layout Standard
The mutual information of a single word can be defined by summing the (fractiona
l) mutual information between a word, and all of it's disjuncts:
\begin_inset Formula 
\[
MI_{word}(w)=\frac{1}{p(w)}\sum_{d}p(w,d)MI_{pair}(w,d)
\]

\end_inset

This is also written in the 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 style, so that, again, the total MI of the entire dataset can be written
 as
\begin_inset Formula 
\[
MI_{cset}=\sum_{w}p(w)MI_{word}(w)
\]

\end_inset

That is, 
\begin_inset Formula $MI_{word}(w)$
\end_inset

 is the fractional contribution of the word to the total MI.
 The fractional MI is very convenient for comparing different words, since
 it factors out the frequency of how often a word is observed: the MI of
 two words with two very different frequencies can be directly compared.
 As can be seen from the graph below, the fractional MI ranges between +3
 and +19 for this dataset.
\end_layout

\begin_layout Standard
The total MI for the dataset is measured to be 7.985 bits.
 
\end_layout

\begin_layout Standard
The distribution can be visualized in two different ways.
 The graph on the left, below
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
This is graphed by sorted-word-mi-hi-p in disjunct-stats.scm 
\end_layout

\end_inset

 shows the ranked MI of the 630 words that have been observed more than
 100 times in this dataset.
 Note that it is a semi-log plot; it is NOT Zipfian.
 Note that the MI seems to decay logarithmically, for a good long ways,
 and then drops off a cliff.
 
\end_layout

\begin_layout Standard
The graph on the right shows the distribution, for all 37413 words, bin-counted
 into 100 bins.
 Unfortunately, this sampling remains small and noisy, so the suggestion
 of a log-linear decay to both left and right is hidden by rather noisy
 spikes.
 The combs on the far right are again the same combs as noted in the last
 figure in section 
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"

\end_inset

 
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"

\end_inset

.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-word-mi-hi-p.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename ../images/binned-word-mi.eps
	width 50col%

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
What re the disjuncts like at either end of this distribution? The first
 ten words in the ranking are: "Cao" "Award" "x" "y" "Prime" "we" "per"
 "Division" "League" "Game".
 The word 
\begin_inset Quotes eld
\end_inset

United
\begin_inset Quotes erd
\end_inset

, examined previously, is 86th in this rank.
 Lets look at some of these.
\end_layout

\begin_layout Standard
The word 
\begin_inset Quotes eld
\end_inset

Award
\begin_inset Quotes erd
\end_inset

 has only two disjuncts that were observed more than twice:
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="3" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
 Grammy- for+ Best+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Academy- for+ Best+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
It is already clear from this one example that the hi-MI words will be those
 that take part in idioms, 
\begin_inset Quotes eld
\end_inset

set phrases
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

institutional phrases
\begin_inset Quotes erd
\end_inset

, and that the disjunct identifies the words taking part in the setting.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="5" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct for 
\begin_inset Quotes eld
\end_inset

Prime
\begin_inset Quotes erd
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Minister+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- Minister+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
as- Minister+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Governor-General- Minister+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
Only one disjunct for 
\begin_inset Quotes eld
\end_inset

Division
\begin_inset Quotes erd
\end_inset

 was observed 4 times; those with 3 observations or less suggest that they
 were extracted from tables listing WWII military battles and equipment.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="2" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct for 
\begin_inset Quotes eld
\end_inset

Division
\begin_inset Quotes erd
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
NCAA-
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The disjuncts for 
\begin_inset Quotes eld
\end_inset

League
\begin_inset Quotes erd
\end_inset

 correspond to word-pairs, and these will clearly be high-MI word-pairs.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="4" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct for 
\begin_inset Quotes eld
\end_inset

League
\begin_inset Quotes erd
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Baseball+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Premier-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Justice-
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
This ranking also captures a fair amount of the 
\begin_inset Quotes eld
\end_inset

garbage
\begin_inset Quotes erd
\end_inset

 in Wikipedia (which is the source for the dataset): the disjuncts on 
\begin_inset Quotes eld
\end_inset

x
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

y
\begin_inset Quotes erd
\end_inset

 indicate that these occur almost exclusively in mathematical formulas of
 some sort, involving single-letter variables, plus signs and equal signs.
 The most frequent disjunct on 
\begin_inset Quotes eld
\end_inset

per
\begin_inset Quotes erd
\end_inset

 was observed 5 times, and was 
\begin_inset Quotes eld
\end_inset

100,000+ 100,000+ 100,000+ 100,000+ 100,000+
\begin_inset Quotes erd
\end_inset

 suggesting that it arose in some parse of a table listing.
 None of the disjuncts on 
\begin_inset Quotes eld
\end_inset

we
\begin_inset Quotes erd
\end_inset

 were observed more than 4 times, and seemed to consist of some sort of
 grammatical word-salad.
 This is not surprising: Wikipedia has a severe shortage of pronouns; style
 guidelines prohibit their use.
\end_layout

\begin_layout Standard
At the other extreme, the ten words with the lowest MI are: "The" "a" "to"
 "in" "of" "and" "the" "," ".".
 These are already familiar from previous rankings: they occur with very
 high frequency; the disjunct lists on them will be lengthly variable, diffuse.
 
\end_layout

\begin_layout Subsection*
Mutual information of disjuncts
\end_layout

\begin_layout Standard
Symmetrically, one also has the mutual information of a disjunct, in comparison
 to all of the words it connects to:
\begin_inset Formula 
\[
MI_{disjunct}(d)=\frac{1}{p(*,d)}\sum_{w}p(w,d)MI_{pair}(w,d)
\]

\end_inset

Again, this is presented in the 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 style, so that the total MI of the entire dataset can be written as before:
\begin_inset Formula 
\[
MI=\sum_{d}p(*,d)MI_{disjunct}(d)
\]

\end_inset

The distribution is shown below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Use binned-dj-mi to get this.
 
\end_layout

\end_inset

 The combs on the far left are again the same combs as noted in the last
 figure in section 
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"

\end_inset

 
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"

\end_inset

.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/binned-dj-mi.eps
	width 60col%

\end_inset


\end_layout

\begin_layout Standard
So, at first, this looks much like the word-MI graph, until one realizes
 its really almost a mirror-image.
 Also very notable is that the MI is negative, in all cases! What does this
 mean? In this case, it means that entropy is dominating over any 
\begin_inset Quotes eld
\end_inset

attractive
\begin_inset Quotes erd
\end_inset

 effect: basically, that the various disjuncts are getting used for a very
 large and widespread set of words, and that any one disjunct is not favoring
 any particular word or word-set; indeed, any given disjunct is mostly dis-favor
ing any word or small set of words, and is instead appearing with just about
 all words.
 This is explored a bit more in the next section.
\end_layout

\begin_layout Subsection*
Fractional Entropy
\end_layout

\begin_layout Standard
There is a simpler variant than the mutual information, that is also worth
 understanding: the fractional contribution to the total entropy.
 This is given by the sum 
\begin_inset Formula 
\[
H_{word}(w)=-\frac{1}{p(w)}\sum_{d}p(w,d)\log_{2}p(w,d)
\]

\end_inset

This is written in the 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 style, so that the total entropy of the entire dataset can be written as
\begin_inset Formula 
\[
H_{cset}=\sum_{w}p(w)H_{word}(w)
\]

\end_inset

 Analogously, one also has the fractional contribution of the disjuncts:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
H_{disjunct}(d)=-\frac{1}{p(*,d)}\sum_{w}p(w,d)\log_{2}p(w,d)
\]

\end_inset

where, again, one has that 
\begin_inset Formula 
\[
H_{cset}=\sum_{d}p(d)H_{disjunct}(d)
\]

\end_inset


\end_layout

\begin_layout Standard
The ranked fractional entropy is shown in the left graph below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Generated from sorted-word-ent in disjunct-stats.scm.
\end_layout

\end_inset

 It only shows those words that have been observed 100 times, or more.
 It resembles the graph for the ranked fractional MI, above.
 The graph on the right shows the distribution of the entropy, for all of
 the words.
 This affirms (or explains?) the sharp knee in the graph on the left: the
 knee occurs because almost all words have a large disjunct-entropy.
 This is explained below.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-word-ent.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename ../images/binned-word-ent.eps
	width 50text%

\end_inset


\end_layout

\begin_layout Standard
The top-ranked words (which have been observed 100 times or more) are these:
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Word
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.22
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
possible
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
education
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
lost
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
days
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
children
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
players
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
available
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.12
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
every
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.12
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
remained
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
above
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
This is not a list that has been exposed with other statistics.
 These words are more 
\begin_inset Quotes eld
\end_inset

interesting
\begin_inset Quotes erd
\end_inset

 than any of the previous lists, which seemed to be filled with quirks associate
d with the dataset.
 By contrast, these are 
\begin_inset Quotes eld
\end_inset

normal
\begin_inset Quotes erd
\end_inset

 words: mostly nouns, some modifiers, a verb, a preposition.
 Why these? 
\end_layout

\begin_layout Standard
The answer lies in looking at the disjunct set, shown in the table below.
 All of these words have a large support, but were observed very infrequently.
 For example, 
\begin_inset Quotes eld
\end_inset

possible
\begin_inset Quotes erd
\end_inset

 has only one disjunct that was observed 3 times, 4 that were observed twice,
 and 101 that were observed only once.
 A quick examination shows that many, maybe most, are grammatically reasonable,
 for example: (investigate- routes+) (is- also- for+) (is- only- to+) (made-
 through+) (many- as-) and so on.
 But all of these were observed precisely once.
 The table below illustrates this.
 
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="5">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{2}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{1}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{0}$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.22
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
possible
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
106
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
education
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
100
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
lost
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
100
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
days
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
98
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
children
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
101
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
A bit of notation.
 Recall the definition for the set that supports a word: 
\begin_inset Formula 
\[
(w,*)=\left\{ (w,d)|N(w,d)>0\right\} 
\]

\end_inset

together with the notation 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

 to indicate the size of this set.
 Extend this notation as
\begin_inset Formula 
\[
\left|(w,*)\right|_{k}=\mbox{sizeof }\left\{ (w,d)|N(w,d)>k\right\} 
\]

\end_inset

That is, 
\begin_inset Formula $\left|(w,*)\right|_{k}$
\end_inset

 is the size of the support for when the disjuncts on word 
\begin_inset Formula $w$
\end_inset

 have been seen more than 
\begin_inset Formula $k$
\end_inset

 times.
\end_layout

\begin_layout Standard
From this table, it is now clear why 
\begin_inset Quotes eld
\end_inset

large entropy
\begin_inset Quotes erd
\end_inset

 can be intuitively understood to mean 
\begin_inset Quotes eld
\end_inset

many possibilities
\begin_inset Quotes erd
\end_inset

.
 Each of these words was seen in a very broad setting of possibilities:
 in a sense, the broadest possible.
 Each of these words was observed with a vary large set of different disjuncts,
 and this set was as spread-out as possible: the vast majority of disjuncts
 were observed exactly once.
 
\end_layout

\begin_layout Subsection*
More terms
\end_layout

\begin_layout Standard
Some curious terms show up in relating the fractional mutual information
 to the fractional entropy.
 Expanding out the above summations, one obtains
\begin_inset Formula 
\[
MI_{word}(w)=H_{word}(w)+\log_{2}p(w,*)+\frac{1}{p(w)}\sum_{d}p(w,d)\log_{2}p(*,d)
\]

\end_inset

 The last term is bizarre...
\end_layout

\begin_layout Subsection*
Vertex degrees and hubiness
\end_layout

\begin_layout Standard
Vertex degrees can be defined as the average number of connectors per disjunct.
 In principle, the vertex degree is an excellent indicator of the part of
 speech.
 For example, determiners, adjectives and adverbs typically have a degree
 of one: they have one connector, which is modifying the noun (or verb)
 that they act on.
 By contrast, nouns typically have a degree of two: one connector to attach
 to a verb, another to a modifier, and that's it.
 Verbs have a degree of three: one connector to a subject, one to a direct
 object, a third to an indirect object or a modifier.
 Of course, nouns might have two or more modifiers, or maybe zero modifiers;
 verbs are also quite variable, but the general concept of vertex degree
 is appealing.
 Closely related to this is the idea of 
\begin_inset Quotes eld
\end_inset

hubiness
\begin_inset Quotes erd
\end_inset

, which can be defined as the second moment of the degree.
 
\end_layout

\begin_layout Standard
Thus, its worth looking at this.
 Define the average degree as
\begin_inset Formula 
\[
K(w)=\frac{\sum_{d,c}N(w,d)C(d,c)}{N(w,*)}
\]

\end_inset

This is graphed, below, for all words that have at least 100 observations.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with sorted-avg-connectors in disjunct-stats.scm 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-avg-connectors.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
This graph is unexpected; in part, having an average number of connectors
 that exceed 5 or 6 is intuitively surprising.
 What's happening here? The first ten items in the ranking are: "-" "de"
 "y" ":" "(" ")" "General" "Department" "x" "Act".
 
\end_layout

\begin_layout Standard
Consider 
\begin_inset Quotes eld
\end_inset

de
\begin_inset Quotes erd
\end_inset

.
 There are 12 observations of the disjunct 
\begin_inset Quotes eld
\end_inset

Janeiro+
\begin_inset Quotes erd
\end_inset

.
 There are 9 observations of the disjunct 
\begin_inset Quotes eld
\end_inset

la+
\begin_inset Quotes erd
\end_inset

.
 There are 51 observations of a disjunct that has 117 connectors on it!!
 This starts out as 
\begin_inset Quotes eld
\end_inset

Diego- Francisco- Francisco- Alonso- Carlos- Fernández- Carlos-
\begin_inset Quotes erd
\end_inset

 and ends with 
\begin_inset Quotes eld
\end_inset

Figueroa+ (+ (+ y+
\begin_inset Quotes erd
\end_inset

 suggesting that there were possibly 51 really bad parses of a very long
 table of Spanish kings, which was mistaken for being a single sentence.
 Clearly, its junk; its frequently-occurring junk, which suggests that the
 table was repeatedly included in maybe 51 different Wikipedia pages.
 
\end_layout

\begin_layout Standard
Similarly, 
\begin_inset Quotes eld
\end_inset

Department
\begin_inset Quotes erd
\end_inset

 has 18 observations of a disjunct with 41 connectors on it.
 It starts with 
\begin_inset Quotes eld
\end_inset

Education- Education- Health- Services- Services- Immigration-
\begin_inset Quotes erd
\end_inset

 and ends with 
\begin_inset Quotes eld
\end_inset

Veterans+ of+ Treasury+ Treasury+
\begin_inset Quotes erd
\end_inset

, again suggesting a bad parse of a table mistaken for a sentence, and included
 in 18 different Wikipedia pages.
 
\end_layout

\begin_layout Standard
The list continues in a similar way, for quite a while.
 The green line suggests that if some 30 or so pathological cases are ignored,
 the system settles down to a more respectable behavior.
 Entries 30 through 50 in the rankings are "Bay" "Street" "Island" "of"
 "century" "right" "Game" "Georgian" "or" "a" ";" "near" "Party" "team"
 "law" "Australia" "her" "research" "Church" "east" "Government".
 Notable is a preponderance of capitalized words, suggesting more tables
 of various sorts, and a complete lack of verbs.
 A spot-check of words like 
\begin_inset Quotes eld
\end_inset

team
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

law
\begin_inset Quotes erd
\end_inset

 shows that the pathological behavior continues.
 Several conclusions are possible.
\end_layout

\begin_layout Standard
One conclusion is that there is a severe shortage of verbs in Wikipedia
 articles, and this makes sense: its primarily descriptive, rather than
 active: running, jumping, hitting, putting, mixing, giving, setting are
 not the kinds of verbs that are required to describe a typical encyclopedia
 topic.
 
\end_layout

\begin_layout Standard
Another conclusion is that perhaps the number of observations of pairs are
 insufficient to get deep, reliable MST parsing.
 Junk links get used because there were not enough appropriate word-pairs
 seen to give a good-quality MST parse.
 A related conclusion is that the connector-set dataset is also too thin:
 The grammatically reasonable connectors are observed not even a few dozen
 times, barely pushing them out of the noise-floor of onesie-twosie observations
 of junk.
 
\end_layout

\begin_layout Standard
So: bigger datasets, and an urgent need for non-Wikipedia content.
 Fiction, and presumably teen fiction should be filled with the kinds of
 active verbs describing human motions and actions, and should be absent
 of tables and lists masquerading as sentences.
\end_layout

\begin_layout Subsection*
Hubiness
\end_layout

\begin_layout Standard
Similar to the above, hubiness can be defined as the second moment of the
 connector count:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
hub(w)=\frac{\sum_{d,c}N(w,d)C^{2}(d,c)}{N(w,*)}-K^{2}(w)
\]

\end_inset

Given the earlier Zipfian results on the average degree 
\begin_inset Formula $K(w)$
\end_inset

, it should be no surprise that a ranked listing of words by hubiness is
 very nearly identical to the listing for average degree.
 This is, after all, what the scale-free nature of the Zipfian distribution
 really means.
 Not only is the ranking nearly the same, but one also has the approximate
 equality 
\begin_inset Formula $hub(w)\approx2K(w)$
\end_inset

 to some ten or twenty percent.
 
\end_layout

\begin_layout Subsection*
Disjunct Cosine Similarity
\end_layout

\begin_layout Standard
The cosine similarity between two vectors is simply their inner product.
 In this case, given two words 
\begin_inset Formula $w_{1}$
\end_inset

 and 
\begin_inset Formula $w_{2}$
\end_inset

, it is given by 
\begin_inset Formula 
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}N(w_{1},d)N(w_{2},d)}{\mbox{len}(w_{1})\mbox{len}(w_{2})}
\]

\end_inset

where 
\begin_inset Formula $\mbox{len}(w)$
\end_inset

 is the root-mean-square length (Euclidean length) of the connector-set
 vector:
\begin_inset Formula 
\[
\mbox{len}(w)=\sqrt{\sum_{d}N^{2}(w,d)}
\]

\end_inset


\end_layout

\begin_layout Standard
The current dataset being analyzed contains 1985 words whose length is greater
 than 8; the ranking-by-length was already shown up above, in a previous
 graph.
 The similarity between all pairs of these was computed; this resulted in
 15808 pairs with a cosine similarity of greater than 0.5.
 These can be sorted and ranked.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
See the good-sims and ranked-sims in disjunct-stats.scm file.
\end_layout

\end_inset

 They are shown below.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/ranked-sims.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Well, that's new! The similarity ranking is very well fit by a cubic! In
 the above, the green line is given by 
\begin_inset Formula $1-\log^{3}(rank)/1800$
\end_inset

 and it fits the tailing end of the distribution so well, that the red data
 line is almost completely hidden under the green line!
\end_layout

\begin_layout Standard
The graph below shows the distribution of cosine similarity.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Produced by binned-good-sims.
 
\end_layout

\end_inset

 There are 5544 words for which 
\begin_inset Formula $4<\mbox{len}(w)$
\end_inset

.
 This shows the distribution of the 2.17 million word-pairs formed from these
 words, and for which 
\begin_inset Formula $0.1\le\mbox{sim}(w_{1},w_{2})$
\end_inset

.
 The eyeballed fit is for 
\begin_inset Formula $\exp(-8\mbox{sim})$
\end_inset

.
 
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename ../images/binned-sims.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
The top-ten similarity pairs are: 'Poir ..
 Perls' 'Poir ..
 Sevan' 'Sevan ..
 Perls' 'Sevan ..
 Ruins' 'Ruins ..
 Perls' 'Poir ..
 Ruins' 'Gongora ..
 Perls' 'Poir ..
 Gongora' 'Gongora ..
 Sevan' 'Gongora ..
 Ruins' 'Poir ..
 Gag'.
 The first three all have a cosine of 1.0, and the rest in the 0.999 range.
 The three words 
\begin_inset Quotes eld
\end_inset

Poir
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

Perls
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

Sevan
\begin_inset Quotes erd
\end_inset

 all have exactly one disjunct, containing exactly one connector, linking
 them to a period, presumably at the end of a sentence.
 Its observed a few dozen times, and that's it.
 
\begin_inset Quotes eld
\end_inset

Gongora
\begin_inset Quotes erd
\end_inset

 has three disjuncts; (.+) is observed 56 times, (,+) is observed twice and
 (,+ .+) is observed twice.
 So, yes, all these words have been discovered to behave in a grammatically
 similar fashion, however, this behavior seems quite boring: links to a
 period.
\end_layout

\begin_layout Standard
The next 300 entries are all links between capitalized words, most of them
 to the words above.
 After that, the list is just barely slightly more varied, with e.g.
 
\begin_inset Quotes eld
\end_inset

tonight
\begin_inset Quotes erd
\end_inset

 showing up occasionally, presumably because there are many sentences that
 end with the word 
\begin_inset Quotes eld
\end_inset

tonight
\begin_inset Quotes erd
\end_inset

, followed by a period.
 There are very few links to lower-case words, but the ones that show up
 are..
 interesting.
 The table below gives some hand-picked examples.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="14" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
cosine
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word-pair
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
504
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.868
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 from'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
633
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.855
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 at'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
637
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.854
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'In ..
 By'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
643
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.854
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 After'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
740
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.846
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'canal ..
 swamp'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
754
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.843
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'at ..
 At'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
757
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.843
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 In
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
818
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.838
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 on'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
840
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.836
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'creation ..
 existence'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
878
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.833
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'e.g ..
 i.e'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
965
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.826
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'and ..
 but'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
969
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.826
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 With'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
971
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.826
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 inside'
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
This table is surprising in several different ways.
 First, that the first entry in which two words are both not capitalized
 does not occur until about 3% into the list.
 Next, that this entry, and most of the others, involve prepositions! Not
 adjectives, adverbs, verbs; although we noted already that the source text
 is verb-poor.
 It is also pleasantly surprising that all of these equivalences appear
 to be correct, including the equivalence of upper-case prepositions that
 start sentences, with lower-case ones that appear mid-sentence.
 The identification of e.g.
 with i.e.
 seems like a bonus, as does the 'and ..
 but' equivalence.
 
\end_layout

\begin_layout Standard
Lets look at this gift-horse in the mouth.
 The equivalence 'canal ..
 swamp' is rather pathetic.
 So, 'canal' was observed with 68 different, unique disjuncts, but most
 have a count of only 1 or 2.
 The top-ranked disjunct on 'canal' is (the-) which is seen 20 times, followed
 by (the- .+) 8 times.
 Similarly, 'swamp' has 18 distinct disjuncts attached, most with counts
 of 1 or 2; the top-ranked disjunct is (the-) with 14 counts, and (the-
 .+) 3 times.
 So, yes, these are both nouns, evidenced by the connection to the word
 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

, but the evidence is otherwise very weak.
 
\end_layout

\begin_layout Standard
Likewise, 'creation' gets the disjunct (the- of+) 18 times, and no other
 disjuncts observed more than twice.
 'existence' gets the same disjunct 10 times, and no other is observed more
 than 3 times.
 In total, 'creation' had 22 disjuncts on it, while 'exstence' had 26, with
 almost all having counts of 1 or 2.
 Again, the evidence is thin, with one notable point: 'creation' is not
 at all like 'canal', in that 'canal' has zero counts of (the- of+) while
 'creation' has only one count of (the-).
 As nouns, they are actually quite different, even based on the thin evidence
 available.
 
\end_layout

\begin_layout Standard
The equivalence of 'in ..
 from' is much more interesting.
 The preposition 'in' has a total of 8338 different unique disjuncts --
 this is huge, compared to the nouns above.
 Likewise, 'from' has 2185 different disjuncts observed.
 The top-ranked disjuncts for each are shown in the tables below.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="8" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in' disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
627
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
75
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
61
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
which+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
49
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the+ the+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
36
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
used-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
his+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
order+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset

 
\begin_inset space \qquad{}
\end_inset

 
\begin_inset Tabular
<lyxtabular version="3" rows="8" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'from' disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
155
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
:+ :+ :+ ..
 0+ 0+ 0+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
derived-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
until+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
degree-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
his+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
The similarity of these two are driven primarily by the word-pairs 
\begin_inset Quotes eld
\end_inset

in the
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

from the
\begin_inset Quotes erd
\end_inset

, followed by 
\begin_inset Quotes eld
\end_inset

in a
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

from a
\begin_inset Quotes erd
\end_inset

.
 Notable also is the common appearance of the pronoun 
\begin_inset Quotes eld
\end_inset

his
\begin_inset Quotes erd
\end_inset

, with 
\begin_inset Quotes eld
\end_inset

in his
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

from his
\begin_inset Quotes erd
\end_inset

.
 The second disjunct on 'from' appears to be some sort of garbage, presumably
 due to the parse of a list of numeric values of some kind.
 
\end_layout

\begin_layout Standard
Just as the case with nouns, the evidence for the similarity for these two
 prepositions seems quite thin.
 The surprise is that, despite this, the identification does seem correct.
\end_layout

\begin_layout Standard
At any rate, the dataset seems a bit thin, but we already knew this; its
 based on a fairly small sample.
\end_layout

\begin_layout Subsection*
Pair Cosine Similarity
\end_layout

\begin_layout Standard
To correctly gauge the disjunct-based cosine similarity, it should be contrasted
 against the simpler pair-based cosine similarity.
 This is given by 
\begin_inset Formula 
\[
\mbox{sim}_{pair}(w_{1},w_{2})=\frac{\sum_{w}N_{pair}(w_{1},w)N_{pair}(w,w_{2})}{\mbox{len}(w_{1})\mbox{len}(w_{2})}
\]

\end_inset

where 
\begin_inset Formula $\mbox{len}(w)$
\end_inset

 is the root-mean-square length (Euclidean length) of the pair vector:
\begin_inset Formula 
\[
\mbox{len}(w)=\sqrt{\sum_{v}N_{pair}(w,v)N_{pair}(v,w)}
\]

\end_inset

and 
\begin_inset Formula $N_{pair}(w,v)$
\end_inset

 is the count of having observed the ordered word-pair 
\begin_inset Formula $(w,v)$
\end_inset

.
 Equivalently, writing the normalized frequency of observing a word pair
 as 
\begin_inset Formula $p(w,v)=N(w,v)/N(*,*)$
\end_inset

, this similarity can be written in the form 
\begin_inset Formula 
\[
\mbox{sim}_{pair}(w_{1},w_{2})=\frac{\sum_{w}p(w_{1},w)p(w,w_{2})}{\sqrt{\sum_{v}p(w_{1},v)p(v,w_{1})}\sqrt{\sum_{v}p(w_{2},v)p(v,w_{2})}}
\]

\end_inset

Note that this similarity measure is NOT symmetric: 
\begin_inset Formula $\mbox{sim}(w_{1},w_{2})\ne\mbox{sim}(w_{2},w_{1})$
\end_inset

.
 This is because it's built out of a manifestly non-symmetric count: 
\begin_inset Formula $p(w,v)\ne p(v,w)$
\end_inset

 and should really be written as 
\begin_inset Formula $p(w,v)=p(R;w,v)$
\end_inset

 with the relation 
\begin_inset Formula $R$
\end_inset

 encompassing all of the constraints of pair-wise word relationships (including,
 for example, that the pair might have been extracted from a random planar
 tree parse).
 Of course, one could construct a symmetrized similarity measure.
\end_layout

\begin_layout Standard
The point here is that this measure treats words as similar with they link,
 pair-wise, to the same kinds of words, with the same kinds of frequencies.
 This is not unlike the similarity that the disjunct-cosine is measuring,
 except that the disjunct carries additional grammatical information with
 it: it captures more complex relationships between the words in a sentence.
\end_layout

\begin_layout Subsection*
Cosine Information
\end_layout

\begin_layout Standard
The cosine similarity was defined as 
\begin_inset Formula 
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}N(w_{1},d)N(w_{2},d)}{\sqrt{\sum_{d}N^{2}(w_{1},d)}\sqrt{\sum_{d}N^{2}(w_{2},d)}}
\]

\end_inset

which, after dividing by 
\begin_inset Formula $N(*,*)$
\end_inset

 so that 
\begin_inset Formula $p(w,d)=N(w,d)/N(*,*)$
\end_inset

, gives the equivalent expression 
\begin_inset Formula 
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}p(w_{1},d)p(w_{2},d)}{\sqrt{\sum_{d}p^{2}(w_{1},d)}\sqrt{\sum_{d}p^{2}(w_{2},d)}}
\]

\end_inset

Comparing this to the expression for mutual information suggests that using
 the vector support, instead of the vector length, could be interesting.
 In particular, these might be interesting:
\begin_inset Formula 
\[
\mbox{com}(w_{1},w_{2})=-\log_{2}\frac{\sum_{d}p(w_{1},d)p(w_{2},d)}{p(w_{1},*)p(w_{2},*)}
\]

\end_inset

 and 
\begin_inset Formula 
\[
\mbox{mim}(w,d)=-\log_{2}\frac{p(w,d)}{\sum_{w,d}p^{2}(w,d)}
\]

\end_inset


\end_layout

\begin_layout Standard
I don't know what to call these; the first seems to be some kind of 
\begin_inset Quotes eld
\end_inset

cosine information
\begin_inset Quotes erd
\end_inset

, the second, some sort of 
\begin_inset Quotes eld
\end_inset

mutual length
\begin_inset Quotes erd
\end_inset

 device.
\end_layout

\begin_layout Subsection*
Quality Cosine
\end_layout

\begin_layout Standard
More possibilities exist.
 The motivation for using cosine similarity is to find pairs of words that
 act in a grammatically similar fashion: they are used in the same way,
 with the same kinds of disjuncts.
 However, observational counts are subject to the vicissitudes of the input
 text: perhaps, instead of using vectors where the components are given
 by the frequency, once could instead use components based on, say, the
 
\begin_inset Quotes eld
\end_inset

quality
\begin_inset Quotes erd
\end_inset

 of the disjunct itself.
 A disjunct could be judged as being 
\begin_inset Quotes eld
\end_inset

high quality
\begin_inset Quotes erd
\end_inset

 if 
\begin_inset Formula $mi(w,d)=MI_{pair}(w,d)$
\end_inset

 as defined in equation 
\begin_inset CommandInset ref
LatexCommand vref
reference "eq:mi-word-disjunct"

\end_inset

 is high.
 This motivates a 
\begin_inset Quotes eld
\end_inset

quality cosine
\begin_inset Quotes erd
\end_inset

: 
\begin_inset Formula 
\[
\mbox{qim}(w_{1},w_{2})=\frac{\sum_{d}mi(w_{1},d)mi(w_{2},d)}{\sqrt{\sum_{d}mi^{2}(w_{1},d)}\sqrt{\sum_{d}mi^{2}(w_{2},d)}}
\]

\end_inset

But what if the high quality is based on a pathetically small number of
 observations? Perhaps there should be some observational weighting.
 One can contemplate defining 
\begin_inset Formula $pi(w,d)=p(w,d)mi(w,d)$
\end_inset

 and so the cosine 
\begin_inset Formula 
\[
\mbox{pim}(w_{1},w_{2})=\frac{\sum_{d}pi(w_{1},d)pi(w_{2},d)}{\sqrt{\sum_{d}pi^{2}(w_{1},d)}\sqrt{\sum_{d}pi^{2}(w_{2},d)}}
\]

\end_inset

 The suitability of these different means of judging similarity is not clear.
\end_layout
\begin_layout Section*
The End
\end_layout

\end_body
\end_document