*! version 1.0 march2014

Title

Freqindex -- Generates an index of terms with their frequencies based on the current dataset

Syntax

freqindex [idvar] txtvar
[, options]

similmethod(simfcn)
Specifies the method to decompose the string into grams.
Default is token. Other built-in simfcn are:
bigram, ngram, ngram_circ, soundex and token_soundex.

incmata(mata_array)
Increments an existing index in memory (mata_array)
with the information from the current dataset.

keepmata
Keeps the Mata objects after conclusion (including mata_array).
Default is dropping them. See list below.

nostata
Omits producing a Stata output with the results (which is the default).
It only makes sense to be used in combination with keepmata
and meant for programming purposes when indexing several files.

Description

freqindex indexes each singular term in a given string variable (txtvar)
from the current dataset and computes its frequencies.
As such, it returns a new dataset containing a string variable listing the terms
(named grams) and a numeric variable with the corresponding frequencies (named freq).
Please, note that freqindex is case-sensitive and
it also takes into account any other symbol (as far as Stata does).

freqindex is also a required element of matchit, which uses it to compute weights.
Moreover, freqindex can be used autonomously as a complementary tool for
computing weights based on custom frequencies or frequencies found in other sources.
When using it with matchit you should always specify the same simfcn in both commands.
Check here for an example of how each built-in simfcn
treats strings.

The numeric variable idvar is optional and
has limited use beyond programming purposes.

Options

Options

txtvar
is the required string varname from the current file to be indexed.

similmethod(simfcn)
explicitly declares the method to parse the two string variables into Grams.
Default is token. Other built-in simfcn are: bigram, ngram, soundex and token_soundex.

similmethod(simfcn,arg)
is the alternative syntax when simfcn requires an argument.
This is the case of ngram and ngram_circ,
which allows computing 1-gram, 2-gram, 3-gram, etc. by passing n as an argument.
For instance, sim(ngram,2) is equivalent to sim(bigram).

keepmata
keeps the Mata objects after conclusion (including mata_array).
Default is dropping them.
It is useful when indexing several columns and/or several files.
See an example below.

nostata
omits producing a Stata output with the results (which is the default).
It only makes sense to be used in combination with keepmata.
It is particularly useful when indexing several columns from the same file
(see an example below).

incmata(mata_array)
Increments an existing index in the Mata associative array (mata_array)
with the information from the current dataset.
Please explicitly set keepmata if you want to keep mata_array
after running freqindex.

idvar
is a numeric varname from the current file identifying its observations.
It is optional and of no use beyond Mata programming.
It only makes sense to be used in combination with keepmata.

Examples:

. freqindex mystring

Setting matching method

. freqindex mystring, sim(soundex)

. freqindex mystring, sim(ngram,3)

Incrementing an existing index

. freqindex mystring1, keepm nost

. freqindex mystring2, incm(WGTARRAY) keepm nost

. freqindex mystring3, incm(WGTARRAY) keepm nost

. freqindex mystring4, incm(WGTARRAY) keepm nost

...

. freqindex mystringN, incm(WGTARRAY)

. list

Saved results

freqindex saves the following in Mata

(only if keepmata option is included)

Mata:

IDW colvector of idvar (only if specified)

TXTW colvector of txtvar

WGTARRAY Array of grams->frequencies

Author

Julio D. Raffo