Introduction

The standard procedure for using the ADM is as follows:

AnnotatedText results = annotator.annotate(getInput(inputFilePathname));
int index = 0;
for (Token token : results.getTokens()) {
    outputData.format("token %d:\t%s%n", index, token.getText());
    int aindex = 0;
    List<MorphoAnalysis> analyses = token.getAnalyses();
    if (null != analyses) {
        outputData.format("\tindex\tlemma\tpart-of-speech%n");
        for (MorphoAnalysis ma : analyses) {
            outputData.format("\t%d    %s\t%s%n", aindex, ma.getLemma(),
                     ma.getPartOfSpeech());
             aindex++;
        }
    }
    index++;
}

Use the TokenizerFactory to create a language-specific tokenizer that extracts tokens from a plain text source. Prior to using the factory to create a tokenizer, you use the factory with TokenizerOption to define the root of your RBL installation, as illustrated in the following sample. See the Javadoc for other options you may set.

The Tokenizer uses a word breaker to establish token boundaries and detect sentences. For each token, it also provides offset information, length of the token and a tag. Some tokenizers calculate morphological analysis information as part of the tokenization process, filling in appropriate analysis entries in the token object that they return. For other languages, you use the analyzer described below to return analysis objects for each token.

Create a tokenizer factory

TokenizerFactory factory = new TokenizerFactory();
factory.setOption(TokenizerOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);
factory.setOption(TokenizerOption.licensePath, 
        new File(rootPath, "licenses/rlp-license.xml").getAbsolutePath());

Set tokenization options

factory.setOption(TokenizerOption.nfkcNormalize, "true");

Use the AnalyzerFactory to create a language-specific analyzer. Prior to creating the analyzer, use the factory and AnalyzerOption to define RBL root, as illustrated in the sample below. See the Javadoc for other options you may set.

AnalyzerFactory factory = new AnalyzerFactory();
factory.setOption(AnalyzerOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);
factory.setOption(AnalyzerOption.licensePath, 
        new File(rootPath, "licenses/rlp-license.xml").getAbsolutePath());

Use the Analyzer to return an array of Analysis objects for each token.

For a multithreaded application it is recommended that a pool of annotators be constructed. Annotators must be used on a per-thread basis, and it is desirable to avoid the overhead of creating annotators each time one is required. The settings of annotators cannot be changed after they are built, so if multiple configurations are required they should be stored separately.

RBL integrates easily with multi-threaded architectures, but, to avoid performance penalties, it does not make promiscuous use of locks. Instead, most objects and interfaces in RBL are either read-only, reentrant objects or read-write, per-thread objects. TokenizerFactory and AnalyzerFactory objects are hybrids as they have both thread-safe and per-thread methods. The create methods of these factories are thread-safe because they do not alter any data within the factories, but the setOption, addDynamicUserDictionary, and addUserDefinedDictionary methods are not thread-safe because they do alter data within the factory. The tokenizers and analyzers created by these factories are always meant to be used on a per-thread basis because they are not reentrant and do alter data within the objects. You can use a factory across multiple threads to create objects as long as calls to the factory methods for setting options or adding user dictionaries are synchronized appropriately. The objects created by the factory must each be created and used by only one thread, which need not be the thread initializing the factory.