Customizing entity extraction in Rosette Server

The Rosette Server endpoints are configured by the files found in the /launcher/config/rosapi directory. Be careful when editing any of these files as the endpoints will not work if not configured properly.

The entity extraction and linking parameters are in the file rex-factory-config.yaml.

For details on how Rosette entity extraction works and how to customize it, refer to the Rosette Entity Extractor Application Developers Guide included in the Rosette Server package.

Entity Extraction Parameters

To modify the default configuration of the /entities endpoint, edit the file /launcher/config/rosapi/rex-factory-config.yaml. The file contains the following parameters. To change the value of a parameter, uncomment the parameter and set the new value.

Parameter	Description	Default
`rootDirectory`	A REX root directory contains language models and necessary configuration files.	`${rex-root}`
`rblRootDirectory`	The directory containing the RBL root for REX to use.	`${rex-root}/rbl-je`
`rejectGazetteers`	Additional gazetter files used to reject entities for the given language.	`null`
`snapToTokenBoundaries`	Regular expressions and gazetteers may be configured to match tokens partially independent from token boundaries. If true, reported offsets correspond to token boundaries.	`true`
`rejectRegularExpressionSets`	Additional regex files used to reject entities.	`null`
`allowPartialGazetteerMatches`	The option to allow partial gazetteer matches. For the purposes of this setting, a partial match is one that does not line up with token boundaries as determined by the internal tokenizer. This only applies to accept gazetteers.	`false`
`kbs`	Custom list of Knowledge Bases for the linker, in order of priority	`null`
`maxEntityTokens`	The maximum number of tokens allowed in an entity returned by Statistical Entity Extractor. Entity Redactor discards entities from Statistical Entity Extractor with more than this number of tokens.	`8`
`customProcessors`	Custom processors to add to annotators.	`null`
`acceptRegularExpressionSets`	Additional files used to produce regex entities.	`null`
`calculateConfidence`	If true, entity confidence values are calculated. Can be overridden by specifying `calculateConfidence` in the API call.	`false`
`joinerRuleFiles`	File containing additional joiner rules.	`null`
`excludedEntityTypes`	Entity types to be excluded from extraction.	`null`
`dataOverlayDirectory`	An overlay directory is a directory shaped like the `data` directory. REX will look for files in both the overlay directory and the root directory, using files from both locations. However, if a file exists in both places (as identified by its path relative to the overlay or root data directory), REX prefers the version in the overlay directory. If REX finds a zero-length file in the overlay directory, it ignores both that file and any corresponding file in the root data directory.	`null`
`confidenceThreshold`	The confidence value threshold below which entities extracted by the statistical processor are ignored.	`-1.0`
`resolvePronouns`	When true, resolve pronouns to person entities.	`false`
`statisticalModels`	Additional files used to produce statistical entities for the given language. You may pass multiple statistical models. The parameter should be formatted in trios of values specfying language, case-sensitivity and the model file, separated by commas. Case-sensitivity can be `automatic`, `caseInsensitive` or `caseSensitive`. For example, setting two models for case-sensitive English and Japanese might look like : `eng,caseSensitive,english-model.bin,jpn,automatic,japanese-model.bin`	`null`
`acceptGazetteers`	Additional gazetteer files used to produce entities for the given language.	`null`
`linkEntities`	The option to link mentions to knowledge base entities with disambiguation model. Enabling this option also enables `calculateConfidence`.	`false`
`caseSensitivity`	The capitalization (aka 'case') used in the input texts. Processing standard documents requires caseSensitive, which is the default. Documents with all-caps, no-caps or headline capitalization may yield higher accuracy if processed with the caseInsensitive value. Can be `automatic`, `caseSensitive` or `caseInsensitive`	`caseSensitive`
`maxResolvedEntities`	The maximum number of entities for in-document coreference resolution (a.k.a. chaining).	`2000`
`calculateSalience`	If true, entity chain salience values are calculated. Can be overridden by specifying `calculateSalience` in the API call.	`false`
`retainSocialMediaSymbols`	The option to retain social media symbols ('@' and '#') in normalized output	`false`
`statSalienceMode`	An option to calculate entity-chain salience with statistical-based calculation (returns 0 or 1) or simple calculation (returns score between 0 and 1)	`true`
`customProcessorClasses`	Register a custom processor class.	`null`
`keepEntitiesInInput`	The option to keep existing annotated text entities.	`false`
`redactorPreferLength`	The option to prefer length over weights during redaction. If true, the redactor will always choose a longer entity over a shorter one if the two overlap, regardless of their user-defined weights. In this case, if the lengths are the same, then weight is used to disambiguate the entities. If false, the redactor will choose the higher weighted entity when two overlap, regardless of the length of the entity string. In this case, if the weights are the same, then the redactor will choose the longer of the two entities.	`true`
`useDefaultConfidence`	The option to assign default confidence value 1.0 to non-statistical entities instead of null.	`false`
`linkingConfidenceThreshold`	The confidence value threshold below which linking results by the kbLinker processor are ignored.	`-1.0`
`indocType`	An option for document entity resolution (also known as entity chaining). Valid values are: `HIGH`, `STANDARD`, `STANDARD_MINUS` or `NULL`	`STANDARD`
Default processors: `acceptGazetteer, acceptRegex, rejectGazetteer, rejectRegex, statistical indocCoref, redactor, joiner processors`	List the set of active processors for an entity extraction run. All processors are active by default. This method provides a way to turn off selected processors. The order of the processors cannot be changed. Note that turning off redactor can cause overlapping and unsorted entities to be returned.	`null`
`supplementalRegularExpressionPaths`	The option to add supplemental regex files, usually for entity types that are excluded by #default. The supplemental regex files are located at `data/regex/<lang>/accept/supplemental` and are not used unless specified.	`null`
`structuredRegionsProcessingType`	Configures how structured regions will be processed. It has three values: `none`, `nerModel`, and `nameClassifier`.	`none`
`regexCurrencySplit`	Determines if money values should be extracted as `MONEY` or `CURRENCY_AMT` and `CURRENCY_TYPE`. If true, REX tries to extract `CURRENCY` instead of `MONEY`.	`false`

Overlay Data Directory

If your project has a set of unique data files that you would like to keep separate from other data files, you can put them in their own directory, also known as an overlay directory. This is an additional data directory, which takes priority over the default REX data directory.

The overlay directory must have the same directory tree as the provided data directory. If an overlay directory is set, REX searches both it and the default data directory.

If a file exists in both places, the version in the overlay directory is used.
If there is an empty file in the overlay directory, REX will ignore the corresponding file in the default data directory.
If there is no file in the overlay directory, REX will use the file in the default directory.

To specify the overlay directory use:

Create an overlay directory:
```
<install-directory>/my-data
```
Add the overlay directory to the rex-factory-config.yaml file:
```
dataOverlayDirectory:
  <install-directory>/my-data
```

Example 5. Turn Off a Specific Language Gazetteer

Create an overlay directory:
Add an empty file (gaz-LE.bin ) to the overlay directory:
```
my-data/gazetteer/eng/accept/gaz-LE.bin
```
Add the overlay directory to the rex-factory-config.yaml file:
```
dataOverlayDirectory:
  <install-directory>/my-data
```

The default English gazetteer will not be used in calls.

Example 6. Use a Custom German Reject Gazetteer

In the above example, add a reject gazetteer file:

my-data/gazetter/deu/reject/reject-names.txt

Default configuration

The default configuration of the /entities endpoint uses the same default values as the REX Java SDK. It is optimized to be more performance-oriented, with fewer options enabled, than the configuration of Rosette Cloud. Rosette Cloud is configured to provide a fully-functional demonstration environment. ^[1]

Table 7. Server and Cloud Default Parameters

Feature	Rosette Server	Parameter Setting in Rosette Server	Rosette Cloud
Entity Linking	false (disabled)	`linkEntities:false`	true (enabled)
Regular Expression Files	no files are loaded	`supplementalRegularExpressionPaths null`	all files are loaded
Pronominal Resolution	false (disabled)	`resolvePronouns: false`	true (enabled)
Case Sensitivity	case-sensitive	`caseSensitivity: caseSensitive`	automatic
Structured Regions	none	`structuredRegionsProcessingType: none`	`nerModel`

Entity linking

Text that refers to an entity is called an entity mention, such as “Bill Clinton” and “William Jefferson Clinton”. Rosette connects these two entity mentions with entity linking, since they refer to the same real-world PERSON entity. Linking helps establish the identity of the entity by disambiguating common names and matching a variety of names, such as nicknames and formal titles, with an entity ID.

Rosette uses the Wikidata knowledge base as a base to link Person, Location, and Organization entities. If the entity exists in Wikidata, then Rosette returns the Wikidata QID, such as Q1 for the Universe. If Rosette cannot link the entity, then it creates a placeholder temporary (“T”) entity ID to link mentions of the same entity in the document. However, the TID may be different across documents for the same entity.

Rosette supports linking to other knowledge bases, specifically the DBpedia ontology and the Thomson Reuters PermID. You can also link to one or more custom knowledge bases. Contact BasisTech Support for more information on using the Field Training Kit (FTK) to add your own linking knowledge base.

Entity linking in Rosette Server is off by default, to improve call speed. When entity linking is turned off, Rosette returns the entities with a TID.

You can enable entity linking in Rosette Server for a single call or as a system default in your environment.

Per-call: add {"options": {"linkEntities": true}} to your call.

Default: edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

#The option to link mentions to knowledge base entities with #disambiguation model.
#Enabling this option also enables calculateConfidence.
linkEntities: true

By default, linking to DBpedia is turned off. To turn it on:

Enable entity linking
Add {"options": {"includeDBpediaTypes": true}} to the call

The list of knowledge bases can be customized in the rex-factory-config.yaml file with the kbs parameter, which takes a List of Paths to knowledge bases.

kbs:
    - /customKBs/kb1
    - /customKBs/kb2
    - /rosette/server/roots/rex/7.44.1.c62.2/data/flinx/data/kb/basis

Note

Setting the list of knowledge bases completely overwrites the list of knowledge bases the linker uses. If you want the default Wikidata knowledge base to be included, it must be on the list of knowledge bases.

Loading regex files

By default, the supplemental regex files are not loaded when the entity extraction endpoint is loaded. To load either the provided supplemental files or new files, edit the /launcher/config/rosapi/rex-factory-config.yaml file adding the supplementalRegularExpressionPaths statements to the file, as show below.

Tip

The files are named following the pattern: data/regex/<lang>/accept/regexes.xml, where <lang> is either the ISO 639-3 language code for the supported language, or xxx for all or any languages.

#The option to add supplemental regex files, usually for entity types that are excluded by
#default. The supplemental regex files are located at data/regex/<lang>/accept/supplemental and
#are not used unless specified.
supplementalRegularExpressionPaths:
- "${rex-root}/data/regex/eng/accept/supplemental/date-regexes.xml"
- "${rex-root}/data/regex/eng/accept/supplemental/geo-regexes.xml"

Pronominal resolution

Entity extraction can try to resolve pronouns with their antecedent entities. For example, in the sentences:

John Smith lives in Boston. He is originally from New York

pronominal resolution would resolve he with John Smith. By default, pronominal resolution is disabled.

To enable it, edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

#The option to resolve pronouns to person entities.
resolvePronouns: true

Case sensitivity

Case sensitivity refers to the capitalization (aka 'case') used in the input texts. Entity extraction can use case to help identify named entities (such as proper nouns) in documents.

Valid values for caseSensitivity

caseSensitive: (default) Case found in standard documents, those in which case follows grammar for the most part.
caseInsensitive: Used for documents with all-caps, no-caps, or headline capitalization. These are documents in which capitalization is not a good indicator for named entities.
automatic: Rosette detects the case from the input model and chooses an appropriate model to use.

To change the default case sensitivity, edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

#The capitalization (aka 'case') used in the input texts. Processing standard documents
#requires caseSensitive, which is the default. Documents with all-caps, no-caps or headline
#capitalization may yield higher accuracy if processed with the caseInsensitive value.
caseSensitivity: automatic

Articles in this section

Entity Extraction Parameters

Overlay Data Directory

Default configuration

Entity linking

Note

Loading regex files

Tip

Pronominal resolution

Case sensitivity

Comments

Articles in this section

Entity Extraction Parameters

Overlay Data Directory

Default configuration

Entity linking

Note

Loading regex files

Tip

Pronominal resolution

Case sensitivity

Related articles