Introduction

The standard procedure for using the ADM is as follows:

AnnotatedText results = annotator.annotate(getInput(inputFilePathname));
int index = 0;
for (Token token : results.getTokens()) {
    outputData.format("token %d:\t%s%n", index, token.getText());
    int aindex = 0;
    List<MorphoAnalysis> analyses = token.getAnalyses();
    if (null != analyses) {
        outputData.format("\tindex\tlemma\tpart-of-speech%n");
        for (MorphoAnalysis ma : analyses) {
            outputData.format("\t%d    %s\t%s%n", aindex, ma.getLemma(),
                     ma.getPartOfSpeech());
             aindex++;
        }
    }
    index++;
}

Use BaseLinguisticsFactory#createTokenizer to create a language-specific tokenizer that extracts tokens from a plain text source. Prior to using the factory to create a tokenizer, you use the factory with BaseLinguisticsOption to define the root of your RBL installation, as illustrated in the following sample. See the Javadoc for other options you may set.

The Tokenizer uses a word breaker to establish token boundaries and detect sentences. For each token, it also provides offset information, length of the token and a tag. Some tokenizers calculate morphological analysis information as part of the tokenization process, filling in appropriate analysis entries in the token object that they return. For other languages, you use the analyzer described below to return analysis objects for each token.

Create a factory

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);

Set tokenization options

factory.setOption(BaseLinguisticsOption.nfkcNormalize, "true");

Create the tokenizer

Tokenizer tokenizer = factory.createTokenizer();

Use the BaseLinguisticsFactory#createAnalyzer to create a language-specific analyzer. Prior to creating the analyzer, use the factory and BaseLinguisticsOption to define RBL root, as illustrated in the sample below. See the Javadoc for other options you may set.

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);
factory.setOption(BaseLinguisticsOption.licensePath, 
        new File(rootPath, "licenses/rlp-license.xml").getAbsolutePath());
Analyzer analyzer = factory.createAnalyzer();

Use the Analyzer to return an array of Analysis objects for each token.

For a multithreaded application it is recommended that a pool of annotators be constructed. Annotators must be used on a per-thread basis, and it is desirable to avoid the overhead of creating annotators each time one is required. The settings of annotators cannot be changed after they are built, so if multiple configurations are required they should be stored separately.

RBL integrates easily with multi-threaded architectures, but, to avoid performance penalties, it does not make promiscuous use of locks. Instead, most objects and interfaces in RBL are either read-only, reentrant objects or read-write, per-thread objects. BaseLinguisticsFactory objects are hybrids, as they have both thread-safe and per-thread methods. The create methods of this factory are thread-safe because they do not alter any data within the factories, but the setOption, user dictionary, and dynamic user dictionary methods are not thread-safe because they do alter data within the factory. The tokenizers and analyzers created by this factory are always meant to be used on a per-thread basis because they are not reentrant and do alter data within the objects. You can use a factory across multiple threads to create objects as long as calls to the factory methods for setting options or adding user dictionaries are synchronized appropriately. The objects created by the factory must each be created and used by only one thread, which need not be the thread initializing the factory.