Using the RBL-JE Core Classes

This chapter introduces two RBL-JE factories, the objects they create, and the RBL-JE command line driver. For the API, consult the Javadoc in this package.

Use TokenizerFactory to create a language-specific Tokenizer that extracts tokens from a plain text source. Prior to using the factory to create a tokenizer, you use the factory with TokenizerOption to define the root of your RBL-JE installation and your license, as illustrated in the following sample. See the Javadoc for other options you may set.

The Tokenizer uses a word breaker to establish token boundaries and detect sentences. For each token, it also provides offset information, length of the token and a tag. Some tokenizers calculate morphological analysis information as part of the tokenization process, filling in appropriate Analysis entries in the Token object that they return. For other languages, you use the Analyzer described below to return Analysis objects for each token.

Here is a sample:

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rbl-je/Tokenize.java

Use the AnalyzerFactory to create a language-specific Analyzer . Prior to creating the analyzer, use the factory and AnalyzerOption to define RBL-JE root and your license file, as illustrated in the sample below. See the Javadoc for other options you may set.

AnalyzerFactory factory = new AnalyzerFactory();
factory.setOption(AnalyzerOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);
factory.setOption(AnalyzerOption.licensePath, new File(
rootPath, "licenses/rlp-license.xml").getAbsolutePath());

Use the Analyzer to return an array of Analysis objects for each token. Depending on the language, each Analysis object may contain one or more of the following:

For some languages, the Analyzer can disambiguate between multiple Analysis objects and return the disambiguated Analysis object.

Here is an example:

The samples are in rbl-je-<rblversion>/samples/tokenize-analyze. In a Bash shell (Unix) or Command Prompt (Windows), navigate to this directory and use the Ant build script to compile and run both of these samples. Your license (rlp-license.xml) must be in the licenses subdirectory of the RBL-JE installation.

To compile and run both samples, call:

Tokenize tokenizes the sample German document and Analyze provides a disambiguated analysis of each token.

The output appears in two files: deu-tokenized.txt and deu-analyzed.txt. The first file contains a token, Tab, <ALNUM> tag on each line, with a blank line following the end of a sentence.^[4]

The second file contains the token, lemma, part of speech, and compound components (where relevant) on each line. For those languages for which disambiguation is not supported,^[5] there may be multiple rows for each token (the token appearing in the first column), one for each analysis. Here is a fragment with a sentence from deu-analyzed.txt:

TOKEN         LEMMA          POS         COMPOUNDS
-----         -----          ---         ---------
3.11.06       3.11.06        CARD
-             -              PUNCT
Not           not            ADV
und           und            COORD
Elend         Elend          NOUN
in            in             PREP
ihren         ihr            POSDET
Heimatländern Heimatland     NOUN         [Heimat, Land]
lassen        lassen         VVFIN
immer         immer          ADV
mehr          mehr           INDADJ
Afrikaner     Afrikaner      NOUN
die           der            ART
Reise         Reise          NOUN
nach          nach           PREP
Europa        Europa         NOUN
antreten      antreten       VVINF
.             .              SENT
        

To run the samples with sample text in a different language, set the test.language parameter with the ISO 639-3 code for the language. For example to tokenize and analyze the Spanish sample, call

RBLCmd is a general-purpose command line utility for RBL-JE. It provides a simple way to produce RBL-JE output without writing code. It is also useful for ad hoc speed and thread testing.

A Bash shell script (RBLCmd) and Windows script (RBLCmd.bat) for running this utility are in rbl-je-<rblversion>/tools/bin. For more information, see RBLCmd's on-line help, RBLCmd -h. Examples of its use are shown below:

$ echo Hello world. | ./tools/bin/RBLCmd --language eng --rootDirectory .
Token{text=Hello}
MorphoAnalysis{extendedProperties={}, partOfSpeech=ITJ, lemma=hello, raw=hello[+ITJ], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=VI, lemma=hello, raw=hello[+VI], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=VPRES, lemma=hello, raw=hello[+VPRES], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=NOUN, lemma=hello, raw=hello[+NOUN], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=PROP, lemma=Hello, tagSet=BT_ENGLISH}
Token{text=world}
MorphoAnalysis{extendedProperties={}, partOfSpeech=NOUN, lemma=world, raw=world[+NOUN], tagSet=BT_ENGLISH}
Token{text=.}
MorphoAnalysis{extendedProperties={}, partOfSpeech=SENT, lemma=., raw=.[+SENT], tagSet=BT_ENGLISH}
$
$
$ echo 'Hello world! :)' | ./tools/bin/RBLCmd --language eng --rootDirectory . --emoticons
Token{text=Hello}
MorphoAnalysis{extendedProperties={}, partOfSpeech=ITJ, lemma=hello, raw=hello[+ITJ], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=VI, lemma=hello, raw=hello[+VI], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=VPRES, lemma=hello, raw=hello[+VPRES], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=NOUN, lemma=hello, raw=hello[+NOUN], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=PROP, lemma=Hello, tagSet=BT_ENGLISH}
Token{text=world}
MorphoAnalysis{extendedProperties={}, partOfSpeech=NOUN, lemma=world, raw=world[+NOUN], tagSet=BT_ENGLISH}
Token{text=!}
MorphoAnalysis{extendedProperties={}, partOfSpeech=SENT, lemma=!, raw=![+SENT], tagSet=BT_ENGLISH}
Token{text=:)}
MorphoAnalysis{extendedProperties={}, partOfSpeech=EMO, lemma=:), tagSet=BT_LANGUAGE_NEUTRAL}
MorphoAnalysis{extendedProperties={}, partOfSpeech=ADJ, lemma=:), raw=:)[+ADJ], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=NOUN, lemma=:), raw=:)[+NOUN], tagSet=BT_ENGLISH}
MorphoAnalysis{extendedProperties={}, partOfSpeech=PROP, lemma=:), raw=:)[+PROP], tagSet=BT_ENGLISH}