The entity extraction endpoint (https://localhost:8181/rest/v1/entities) comes fully configured to extract entities. This guide explains how to modify the configuration of the extractor for your use case.
General Configuration Options
Handling Structured Regions
The REX statistical model is trained to extract entities from unstructured text, where the model uses the syntactic context in sentences to help identify entities and entity types. But not all data is unstructured. Often input documents contain some sections of structured text, such as tables and lists, along with the unstructured text. Structured text usually does not contain full sentences and is often missing the syntactic context that REX expects. This can lead to noisy results and false positives.
In addition to sentences and token, the Rosette Base Linguistics (RBL) processor identifies structured and unstructured regions. For structured regions, REX disables the statistical processor. The text in structured regions is still processed by the rule-based processors (gazetteers and regexes) and the linker. Additionally, for some languages, another extractor, the name classifier, can extract entities from structured regions of text.
By default, structured regions are processed the same as unstructured regions.
To change how structured text is processed, set structuredRegionsProcessingType in the rex-factory-config.yaml file. You can also set the value in the call as an option. It has three values:
-
none: (default) Disables the statistical/DNN models from processing structured regions. When set to none, REX does not attempt to extract entities from structured regions using the statistical processor or DNN models. The rule-based extractors (gazetteers, regex) and the linker are used to process structured regions.
-
nerModel: Processes the entire document as unstructured text. Structured regions are processed the same as unstructured regions.
-
nameClassifier: Disables the statistical/DNN models from processing structured regions and enables the name classifier on the structured regions.
You can enable the Apache Tika processor to extract lists and tables for contentUri (HTML) input by setting the enableStructuredRegion option to true as the default in the rex-factory-config.yaml file or in the call as an option:
"options": {"enableStructuredRegion": true}
Some structured regions may contain enough syntactic context for the statistical/DNN models to accurately extract entities. You can set a minimum number of tokens required in a structured region to override the structured region processor setting. If the number of tokens in the region exceeds this minimum, the region will be processed with the statistical/DNN models. The default value is 0. With this default, all structured regions are processed as defined by the structuredRegionsProcessingType.
To set the minimum number of tokens, change the value of RegionProcessingSentenceTokensMin in the rosette-factory-config.yaml file.
Fragment Boundary Detector
Note
Disabling the fragment boundary detector will classify the entire text as unstructured. This has a similar effect to setting structuredRegionsProcessingType to nerModel.
REX detects entities within sentences. By default, REX uses a fragment boundary detector to identify structured regions, adding sentence boundaries at tabs, newlines, and multiple whitespace characters (such as 3 or more spaces) in text fragments, such as lists and tables. This enables the detection of multiple entities in text fragments that do not form standard sentences. Consider the following text:
George Washington
John Adams
Thomas Jefferson
Without the fragment boundary detector, the statistical model identifies the preceding text as a single PERSON entity. With the fragment boundary detector, the statistical model identifies three separate PERSON entities.
Turn off the fragment boundary detector in the rex-factory-config.yaml file.
#Regular expressions and gazetteers may be configured to match tokens
#partially independent from token boundaries. If true, reported offsets
#correspond to token boundaries.
snapToTokenBoundaries: false
Note
While the fragment boundary detector improves REX's performance on tables, lists, and other non-prose content, REX is, by design, tuned for prose and may not return high accuracy results on content with significant non-prose elements.
If your project has a set of unique data files that you would like to keep separate from other data files, you can put them in their own directory, also known as an overlay directory. This is an additional data directory, which takes priority over the default REX data directory.
The overlay directory must have the same directory tree as the provided data directory. If an overlay directory is set, REX searches both it and the default data directory.
-
If a file exists in both places, the version in the overlay directory is used.
-
If there is an empty file in the overlay directory, REX will ignore the corresponding file in the default data directory.
-
If there is no file in the overlay directory, REX will use the file in the default directory.
To specify the overlay directory use:
-
Create an overlay directory:
<install-directory>/my-data
-
Add the overlay directory to the rex-factory-config.yaml file:
dataOverlayDirectory:
<install-directory>/my-data
Example 1. Turn Off a Specific Language Gazetteer
-
Create an overlay directory:
-
Add an empty file (gaz-LE.bin ) to the overlay directory:
my-data/gazetteer/eng/accept/gaz-LE.bin
-
Add the overlay directory to the rex-factory-config.yaml file:
dataOverlayDirectory:
<install-directory>/my-data
The default English gazetteer will not be used in calls.
Example 2. Use a Custom German Reject Gazetteer
In the above example, add a reject gazetteer file:
my-data/gazetter/deu/reject/reject-names.txt
REX can return a salience score for each extracted entity. Salience indicates whether the entity is important to the overall scope of the document. Returned salience scores are binary, either 0 (not salient) or 1 (salient). The decision is made according to several parameters, such as frequency, distance from document start, etc. Salience is not calculated by default.
To include the salience in a result for by call, add the option to the request:
"options": {"calculateSalience": true}
Or to get the salience by default, set the calculateSalience parameter to true in the rex-factory-config.yaml file.
#An option to calculate entity-chain salience values.
calculateSalience: true
Retrieving Base Linguistics Configuration
REX internally uses Rosette Base Linguistics (RBL) to analyze the text before processing it. If the user application already uses Base Linguistics for other purposes, it's possible to save processing time and have REX annotate pre-toxenized documents by passing REX's annotator annotate function a tokenized AnnotatedText instance instead of a string. However, if the user's instance of Base Linguistics and REX's internal instance of Base Linguistics are configured differently, REX's results might be affected.
To solve the problem, EntityExtractor provides a getBaseLinguisticsParameters function that returns the set of Rosette Base Linguistics options REX uses internally, given a language. This function should be called after the EntityExtractor has been otherwise configured. It returns an EnumSet of keys to the values REX configures them to.
Tip
REX provides a sample (rex-je-<version>/samples/RBLParametersSample.java) which demonstrates how to retrieve RBL parameters from REX and use RBL directly to process documents before running the REX extractor.