Modifying Entity Extraction Processors

By default, REX uses all the processors. You can select to use a subset of the processors. For example, you can decide to return only entities extracted by statistical analysis.

REX includes the following processors:

The order of execution of the processors is determined internally and cannot be changed. Some processors are prerequisites for other processors. REX will throw an exception if the processor list is missing a required processor.

Edit the rex-factory-config.yaml file, modifying the list of active processors for an entity extraction run.

#List the set of active processors for an entity extraction run.
#All processors are active by default. This method provides a way 
#to turn off selected processors. The order of the processors cannot be changed. 
#Note that turning off redactor can cause overlapping and unsorted 
#entities to be returned.
#Default processors:
#acceptGazetteer,
#acceptRegex,
#rejectGazetteer,
#rejectRegex,
#statistical,
#indocCoref,
#redactor,
#joiner
#
processors: 
   statistical

The statistical processor uses models based on computational linguistics and human-annotated training documents. You can add other statistical models to improve extraction for your use case. Extract new entity types or improving the results of the statistical model requires training a new model with using the field training kit (FTK).

Statistical model based extractions can return confidence scores for each entity. Confidence scores correlate well with precision and may be used for thresholding and removal fo false positives. Confidence is calculated by default if linking is enabled. Otherwise, use the calculateConfidence parameter to enable confidence scores. To set a threshold value, use the confidenceThreshold parameter.

REX has a deep neural network (DNN) model that can be used in place of the statistical model for selected languages. By default, REX uses the statistical models rather than the DNN model. You can customize which model Rosette uses.

The deep neural network processor is using TensorFlow 2.3.1 (Java version 0.2.0). Ubuntu Linux 14.04+, Windows 7+, and MacOS 10.11+ are fully supported, but you should be able to run the processor successfully on other modern Linux flavors as well. To use the processor on platforms which are not otherwise supported, or to improve the speed on supported platforms, you can replace the TensorFlow library shipped with the product with one that’s built from source.

To make use of GPUs, you should download tensorflow-core-platform-gpu and add it to the top of your classpath.

To select which model will be used, set the modelType option in your calls. The default value for modelType is statistical. To enable the deep neural network model, provide DNN for the modelType. Example:

{"content": "your_text_here", "options": {"modelType": "DNN"}}

Currently, REX has DNN models for the following languages:

REX has a name classifier which can be used in place of the statistical model for structured regions. The name classifier is a machine learning model that tries to predict an entity type for an input string. It processes the entire structured region (the input string) as a single entity, predicting a label (PERSON, LOCATION, ORGANIZATION, or NONE) for the string. It works best on tables cells or list items where the entire entry is a single entity. If a structured region contains more text than the entity mention itself, the name classifier will usually label it as NONE.

To enable the name classifier for structured regions, set structuredRegionsProcessingType to nameClassifier in the rex-factory-config.yaml file.

Currently, REX supports the name classifier processor for the following languages:

Each language has its own configuration file, data/name_classifier/<lang>/<lang>_config.yaml, where <lang> is the 3 letter language code. The labelScoreThresholds field determines the chance that a classifier will label a phrase with a given entity type. Lowering the threshold will label more phrases, which will find more true positives, but may also identify more false positives.

To disable an entity type completely, remove or comment out the corresponding entry from the <lang>_config.yaml file. Example:

# labelScoreThresholds
# Set the model score thresholds for each entity type.
# To turn off an entity from the model, comment it out.
# The accuracy of the current ORG model is too low and so it is better to turn it off for now.
labelScoreThresholds:
  PER: 1.2
  LOC: 3.2
#  ORG: 5.2

A gazetteer is a list of exact matches in a predefined closed class. For example, you can use a gazetteer to match all the countries in the world, as there is a precise and unambiguous list of countries. An entry would count as ambiguous if it has multiple possible meanings, such as "Apple", which could be either an ORGANIZATION or a fruit. The gazetteers are very fast at extracting entities. If you are searching for specific words or phrases in your data, a custom gazetteer is a good way to find them quickly.

REX is shipped with default gazetteer files which you can modify, Gazetteer files are located in a subdirectory of the data directory, defined by language using the three-letter ISO-639-3 language code. A directory which applies to all languages, uses xxx for the language code. For example:

<install-directory>/roots/rex-<version>/data/gazetteer/eng/reject/
<install-directory>/roots/rex-<version>/data/gazetteer/xxx/accept/ 

By default, the data files are located in the <install-directory>/roots/rex-<version> directory. If you want your custom files to be in a separate location, use an Overlay Data Directory.

# File: infectious-diseases-gazetteer.txt
#
DISEASE:INFECTIOUS
tuberculosis
e. coli
malaria
influenza

# File: latin-infectious-gazetteer.txt
#
DISEASE:INFECTIOUS
Mycobacterium tuberculosis
Escherichia coli
Plasmodium malariae
Orthomyxoviridae

# File: infectious-bacterial-gazetteer.txt
#
DISEASE:BACTERIAL
Escherichia coli
E. coli
Staphylococcus aureus
Streptococcus pneuminiae
Salmonella

# File: antimicrobial-drugs-gazetteer.txt
#
DRUG:ANTIMICROBIAL
methicillin
vancomycin
macrolide
fluoroquinolone

#An option for document entity resolution (also known as entity chaining).
indocType: NULL

{"language": "zho",
"configuration": {
"entities": { "ANIMAL": [ "狮子", "獅子" ] }
}
}

curl --request POST \
--url http://localhost:8181/rest/v1/entities/configuration/gazetteer/add \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{"language": "xxx", "configuration":{"entities":{ "COMPANY": ["New Corp", "Best Business"]}}}'

curl --request POST \
--url http://localhost:8181/rest/v1/entities/configuration/gazetteer/add \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{"language": "xxx", \
"configuration":{"entities":{ "COMPANY": ["New Corp", "Best Business"]}}, "profileId": "group1"}'

{"language": "xxx", "configuration": {"entities":{ "COMPANY": ["New Corp", "Best Business"] } } } 

curl --request POST \
--url http://localhost:8181/rest/v1/entities/configuration/gazetteer/add \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '@new_companies.json'

Regular expressions (regexes) are used for finding entities which follow a strict pattern with a rigid form and infinite combinations, such as URLs and credit card numbers. In the default REX installation the regex files are:

You can modify these files to add new patterns to extract the same entity type.

To extract new entity types that have predictable patterns, add a new XML regex file, either the language-specific (<lang> ) or generic (xxx) location. REX uses the Tcl regex format for defining the regex patterns.

REX modifies the regex matcher so that \n in a regex expression matches straight new lines (\n\), carriage returns (\r), or a combination of both (\r\n). Regardless of what is matches, offsets and lengths in the result will match the input document.

By default, the data files are located in the <install-directory>/roots/rex-<version> directory. If you want your custom files to be in a separate location, use an Overlay Data Directory.

#The option to add supplemental regex files, usually for entity types that are excluded by
default. The supplemental regex files are located at data/regex/<lang>/accept/supplemental and
are not used unless specified.

supplementalRegularExpressionPaths: 
  - data/regex/eng/accept/supplemental/geo-regexes.xml

The joiner combines adjacent entities into a single entity, based on the joiners rules. REX then returns the single entity.

The configuration file for joining adjacent entities is in data/etc.

The file neredact-config.xml specifies the rules for joining adjacent entities. Adjacent TITLE entities are joined into a single TITLE entity. The joiner elements for joining TITLE and PERSON entities into a PERSON entity are commented out by default.

<neredactconfig>
  <joiners>
    <joiner left='TITLE' right='TITLE' joined='TITLE'/>
<!-- Not joined by default
    <joiner language='eng' left='TITLE' right='PERSON' joined='PERSON'/>
    <joiner language='jpn' left='PERSON' right='TITLE' joined='PERSON'/>
-->
  </joiners>
</neredactconfig>

Rules can optionally specify a language, in which case they will apply only to entities of that specific language. If a language is not specified, the rule will apply for any language.

Entities are considered adjacent if they are separated by no more than 5 whitespace characters.

For example, to join "Barack Obama" and "President" in "Barack Obama, President", the joiner rule is:

<joiner left='PERSON' adjacency-regex=',\s+' right='TITLE' joined='PERSON'/>

The joiner runs before the redactor, as of release 7.46.2. To run the joiner after the redactor, set the parameter runJoinerPostRedactor to true in the rex-factory-config.yaml file.

The redactor determines which entity to choose when multiple mentions for the same entity are extracted. The redactor first chooses longer entity mentions over shorter ones. If the length of the mentions are the same, the redactor uses weightings to select an entity mention.

Different processors can extract overlapping entities. For example, a gazetteer extracts "Newton", Massachusetts as a LOCATION, and the statistical processor extracts "Isaac Newton" as a PERSON. When two processors return the same or overlapping entities, the redactor chooses an entity based on the length of the competing entity strings. By default, a conflict between overlapping entities is resolved in favor of the longer candidate, "Isaac Newton".

The redactor can be configured to set weights by:

    <ne_type>
        <name>IDENTIFIER</name>
        <subtypes>
            <name>EMAIL</name>
            <name>URL</name>
            <name>DOMAIN_NAME</name>
            <name>IP_ADDRESS</name>
            <name>PHONE_NUMBER</name>
            <name>FAX_NUMBER</name>
            <name>PERSONAL_ID_NUM</name>
            <name>CREDIT_CARD_NUM</name>
            <name>MONEY</name>
            <name>PERCENT</name>
            <name>NUMBER</name>
        </subtypes>
        <weight name="statistical" value="9" />
        <weight name="gazetteer" value="10" />
        <weight name="regex" value="10" />
    </ne_type>

        <weight name="statistical" value="1" />
        <weight name="gazetteer" value="10" />
        <weight name="regex" value="5" />

        <weight name="kb-linker:MyKB" value="20" />
        <weight name="kb-linker" value="1" />

Instead of adding entities to extract when matched you can define a list of entities to reject when matched. These are reject gazetteers.

The format of a reject gazetteer is identical to the format of an accept gazetteer except the wildcard (*) is allowed in the entity type. As with accept gazetteers, they are arranged by language.

If, for example, it is for rejecting German entities, put it in data/gazetteer/deu/reject. If it is for rejecting entities in multiple languages, put it in data/gazetteer/xxx/reject.

PERSON
George Watson

*
George Watson

A typical regex is used to identify entities of a specified entity type. You can also define a regex to reject entities; that is whenever the pattern is identified, the entity is rejected as the defined type. Reject regexes follow the same format as accept regexes with the addition that the wildcard (*) is allowed for the entity type.

Place your reject regex files in the following directories:

For example, a file to reject German entities, is named data/regex/deu/reject. Files rejecting entities in multiple languages go in data/regex/xxx/reject.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE regexps PUBLIC "-//basistech.com//DTD RLP Regular Expression Config 7.1//EN"
                          "urn:basistech.com:7.1:rlpregexp.dtd">

<regexps>
  <regexp lang="eng" type="LOCATION">Baltimore</regexp>
</regexps>

Documents often contain multiple references to a single entity. In-document coreference (indoc coref) chains together all mentions to the same entity.

If resolvePronouns is enabled (it is disabled by default), REX will try to resolve pronouns with the corresponding antecedent entities.