Analyzers

For each token and normalized form in the token stream, the analyzer performs a dictionary lookup starting with any user dictionaries followed by the RBL dictionary. During lookup, RBL ignores the context in which the token or normalized form appears.

Once the analyzer has found one or more lemmas in a dictionary, it does not consult additional dictionaries. In other words, if two user dictionaries are specified, and the filter finds a lemma in the first dictionary, it does not consult the second user dictionary or the RBL dictionary.

Unless overridden by an analysis dictionary, the only lemmatization done in Chinese and Thai is number normalization. Other Chinese and Thai tokens' lemmas are equal to their surface forms.

There is no analysis dictionary available for Finnish, Pashto, or Urdu. All other languages are supported.

No dictionary can ever be complete: new words get added to languages, languages change and borrow. So, in general, analysis for each language includes some sort of guessing capability. The job of a guesser is to take a word and to come up with some analysis of it. Whatever facts we generate for a language, those are all possible outputs of a guesser.

In European languages, guessers deliver lemmas and parts of speech. In Korean, guessers provide morphemes, morpheme tags, compound components, and parts of speech.

By default, the analyzer returns any lemma that contains whitespace as multiple lemmas (each with no whitespace). To allow lemmas with whitespace (such as International Business Machines as a lemma for the token IBM) to be placed as such in the token stream, you can create a user analysis dictionary with an entry that defines the lemma. For example:

IBM International[^_]Business[^_]Machines[+PROP]

The analyzer decomposes Chinese, Danish, Dutch, German, Hungarian, Japanese, Korean, Norwegian, and Swedish compounds, returning the lemmas of each of the components.

The lemmas may differ from their surface form in the compound, such that the concatenation of the components is not the same as the original compound (or its lemma). Components are often connected by elements that are present only in the compound form.

For example, the German compound Eingangstüren (entry doors) is made up of two components, Eingang (entry) and Tür (door), and the connecting 's' is not present in the component list. For this input token, the RBL tokenizer and analyzer return the following entries:

Other German examples include letter removal (Rennrad ⇒ rennen + Rad), vowel changes (Mängelliste ⇒ Mangel + Liste), and capitalization changes (Blaugrünalge ⇒ blau + grün + Alge).

Enum Classes:

For some languages, the analyzer can disambiguate between multiple analysis objects and return the disambiguated analysis object. The disambiguate option enables the disambiguator. When true, the disambiguator determines the best analysis for each word given the context in which it appears.

When using an annotator, the disambiguated result is at the head of all possible analyses. The remainder of the list is ordered randomly. When using a tokenizer/analyzer, use the method getSelectedAnalysis to return the disambiguated result.

For all languges except Japanese, disambiguation is enabled by default. For performance reasons, disambiguation is disabled by default for Japanese when using the statistical model.

Enum Classes:

In RBL, each language has its own set of POS tags and a few languages have multiple tag sets. Each tag set is identified by an identifier, which is a value of the TagSet enum. When RBL outputs a POS tag, it also lists the identifier for the tag set it came from. Output from a single language may contain POS tags from multiple tag sets, including the language-neutral set.

POS tags are defined for Arabic, Chinese, Czech, Dutch, English, French, German, Greek, Hebrew, Hungarian, Indonesian, Italian, Japanese, Korean, Malay (Standard), Persian, Polish, Portuguese, Russian, Spanish, Tagalog, and Urdu.

-
  - NUM_VOC
  -
    - { m: \+Total, t: PRON }
    - { s: moc, t: DET }
    - { s: oba, t: DET }
    - { t: NUM }

You can split contractions and return analyses with tokens, lemmas, and POS tags for each constituent. For example, given the English contraction can't, RBL returns analyses for can and for not. To split contractions, set tokenizeContractions to true.

Contractions are defined by contraction rule files. By default, the tokenizer uses the rules in rootDirectory/contractions/contraction-rules-<language>.yaml, where <language> is the language code. RBL comes with contraction rules for English, German, and Portuguese. To add rules for these languages or add rules to support another language, edit the default files or create a custom rule file. The URI for the custom file is defined by customTokenizeContractionRulesUri.

Enum Classes:

For a sample, see rbl-je-<version>/samples/contractions.

-
  - [ "ain't", "VBPRES" ]
  -
    - [ "am", "VBPRES", "be", null ]
    - [ "not", "NOT", "not", null ]
-
  - [ "amn't", "ADJ" ]
  -
    - [ "am", "VBPRES", "be", null ]
    - [ "not", "NOT", "not", null ]
-
  - [ "amn't", "NOUN" ]
  -
    - [ "am", "VBPRES", "be", null ]
    - [ "not", "NOT", "not", null ]