User Dictionaries

All static and dynamic user dictionaries, except for many-to-one normalization dictionaries, are consulted in reverse order of addition. In cases of conflicting information, dictionaries added later take priority over those added earlier. Once a token is found in a user dictionary, RBL stops and will consult neither the remaining user dictionaries nor the RBL dictionary.

Many-to-one normalization dictionaries are consulted in the following order:

Example of non-many-to-one user dictionary priority:

Dictionaries are prioritized in the following order:

dynDict4, statDict3, statDict2, dynDict1

Example of many-to-one normalization user dictionary priority:

Dictionaries are prioritized in the following order:

dynDict4, dynDict1, statDict2, statDict3

The Chinese and Japanese language analyzers load all dictionaries with the user dictionaries loaded at the end of the list. To prioritize the user dictionaries and put them at the front of the list, guaranteeing the matches in the user dictionaries will be used, set the option favorUserDictionary to true.

The following formatting rules apply to user dictionary source files.

Once complete, the source file is compiled into a binary format for use in RBL.

norm1        var1        var2        var3

dictionary.add("norm1", "var1", "var2", "var3");

In the tools/bin directory, RBL includes a shell script for Unix

rbl-build-user-dictionary

and a .bat file for Windows.

rbl-build-user-dictionary.bat

To compile a user dictionary, from the RBL root directory:

tools/bin/rbl-build-user-dictionary -type TYPE_ARGUMENT LANG INPUT_FILE OUTPUT_FILE

where TYPE_ARGUMENT is the dictionary type, LANG is the language code, INPUT_FILE is the pathname of the source file you have created, and OUTPUT_FILE is the pathname of the binary compiled dictionary the tool creates. For example:

tools/bin/rbl-build-user-dictionary -type analysis jpn jpn_lemmadict.txt jpn_lemmadict.bin

The script uses Java to compile the user dictionary. The operation is performed in memory, so you may require more than the default heap size. You can set heap size with the JAVA_OPTS environment variable. For example, to provide 8 GB of heap size, set JAVA_OPTS to -Xmx8g.

Unix shell:

export JAVA_OPTS=-Xmx8g

Windows command prompt:

set JAVA_OPTS=-Xmx8g

The format for a segmentation dictionary source file is very simple. Each word is written on its own line, and that word is guaranteed to be segmented as a single token when seen in the input text, regardless of context. Japanese example:

三菱UFJ銀行
酸素ボンベ

Each entry is a word, followed by a tab and an analysis. The analysis must end with a lemma and a part-of-speech (POS) tag.

word	lemma[+POS]

For those languages for which RBL does not return POS tags, use DUMMY.

Variations. You can provide more than one analysis for a word or more than one version of a word for an analysis.

The following example includes two analyses for "telephone" (noun and verb), and two renditions of "dog" for the same analysis (noun).

telephone	telephone[+NOUN]
telephone	telephone[+VI]
dog	dog[+NOUN]
Dog	dog[+NOUN]               

For some languages, the analysis may include special tags and additional information.

Contracted forms. For English, French, Italian, and Portuguese, ^= is a separator for a contraction or elision.

English example:

doesn't	does[^=]not[+VDPRES]

Multi-Word Analysis. For English, Italian, Spanish, and Dutch, ^_ indicates a space.

English example:

IBM	International[^_]Business[^_]Machines[+PROP]

Compound Boundary. For Danish, Dutch, Norwegian, German, and Swedish, ^# indicates the boundary between elements in a compound word. For Hungarian, the compound boundary tag is ^CB+.

German example:

heimatländern	Heimat[^#]Land[+NOUN]

Compound Linking Element. For German ^/, indicates a compound linking element. For Dutch, use ^//.

German example:

arbeitskreis	Arbeit[^/]s[^#]Kreis[+NOUN]

Derivation Boundary or Separator for Clitics. For Italian, Portuguese, and Spanish, ^| indicates a derivation boundary or separator for clitics.

Spanish example with derivation boundary:

duramente	duro[^|][+ADV]

Italian example with separator for clitics:

farti	fare[^|]tu[+VINF_CLIT]

Japanese Readings and Normalized Forms. For Japanese, [^r] precedes a reading (there may be more than one), and [^n] precedes a normalization. For example:

行わ	行う[^r]オコナワ[+V]
tv	テレビ[^r]テレビ[^n]テレビ[+NC]
アキュムレータ	アキュムレーター[^r]アキュムレータ[^n]アキュムレーター[+NC]

Korean Analysis. A Korean analysis uses a different pattern than the analysis for other languages. The pattern for an analysis in a user Korean dictionary is as follows:

Token	Mor1[/Tag1][^+]Mor2[/Tag2][^+]Mor3[/Tag3]

Where each MorN is a morpheme, consisting of one or more Korean characters, and TagN is the POS tag for that morpheme. [^+] indicates the boundary between morphemes.

Here's an example:

유전자이다	유전자[/NPR][^+]이[/CO][^+]다[/ECS]

If the analysis contains one noun morpheme, that morpheme is the lemma and the POS tag is the POS tag for that morpheme. If more than one of the morphemes are nouns, the lemma is the concatenation of those nouns (a compound). Example:

정보검색	정보[/NNC][^+]검색/[NNC]

Otherwise, the lemma is the first morpheme, and the POS tag is the POS tag associated with that morpheme.

You can override this algorithm for identifying the lemma and/or POS tag in a user dictionary entry by placing [^L]lemma and/or [^P][/Tag] at the end of the analysis. The lemma may or may not correspond to one of the morphemes in the analysis. For example:

유전자이다	유전자[/NNC][^+]이[/CO][^+]다[/ECS][^L]유전[^P][/NPR]

The KoreanAnalysis interface provides access to the morphemes and tags associated with a given token in either the standard Korean dictionary or a user Korean dictionary.

A many-to-one normalization dictionary maps one or more variants to a normalized form. The first value on each line is the normalized form. The remainder of the entries on the line are the variants to be mapped to the normalized form. All values on the line are separated by tabs.

Example:

Use the option normalizationDictionaryPaths to specify the static user normalization dictionaries.

The source file for a Chinese or Japanese user dictionary is UTF-8 encoded (see Valid Characters for Chinese or Japanese User Dictionary Entries). The file may begin with a byte order mark (BOM). Empty lines are ignored. A comment line begins with #. The first line of a Japanese dictionary may begin !DICT_LABEL followed by Tab and an arbitrary string to set the dictionary's name, which is not currently used anywhere.

Each entry in the dictionary source file is a single line:

word Tab POS Tab DecompPattern Tab Reading1,Reading2,...

where word is the entry or surface form, POS is one of the user-dictionary part-of-speech tags listed below, DecompPattern is the decomposition pattern: a comma-delimited list of numbers that specify the number of characters from word to include in each component of the compound (0 for no decomposition), and Reading1,... is a comma-delimited list of one or more transcriptions rendered in Hiragana or Katakana (only applicable to Japanese).

The decomposition pattern and readings are optional, but you must include a decomposition pattern if you include readings. In other words, you must include all elements to include the entry in a reading user dictionary, even though the reading user dictionary does not use the POS tag or decomposition pattern. To include an entry in a segmentation (tokenization) user dictionary, you only need POS tag and an optional decomposition pattern. Keep in mind that those entries that include all elements can be included in both a segmentation (tokenization) user dictionary and a reading user dictionary.

Note: For examples of standard (non-user-dictionary) use of the one and two-letter POS tags in the preceding list, see Japanese POS Tags – BT_JAPANESE_RBLJE_2.

Examples (the last three entries include readings)

!DICT_LABEL New Words 2014
デジタルカメラ NOUN
デジカメ NOUN 0
東京証券取引所 ORGANIZATION 2,2,3
狩野 SURNAME 0
安倍晋三 PERSON 2,2 あべしんぞう
麻垣康三 PERSON 2,2 あさがきこうぞう
商人 NOUN 0 しょうにん,あきんど
        

The POS and decomposition pattern can be in full-width numerals and Roman letters. For example:

東京証券取引所 ｏｒｇａｎｉｚａｔｉｏｎ ２,２,３

The "2,2,3" decomposition pattern instructs the tokenizer to decompose this compound entry into

東京
証券
取引所