Articles in this section

See more

Customizing entity extraction in Rosette Server

Avatar
Jason Stevens
  • Updated
Follow

The Rosette Server endpoints are configured by the files found in the /launcher/config/rosapi directory. Be careful when editing any of these files as the endpoints will not work if not configured properly.

The entity extraction and linking parameters are in the file rex-factory-config.yaml.

For details on how Rosette entity extraction works and how to customize it, refer to the Rosette Entity Extractor Application Developers Guide included in the Rosette Server package.

Entity Extraction Parameters

To modify the default configuration of the /entities endpoint, edit the file /launcher/config/rosapi/rex-factory-config.yaml. The file contains the following parameters. To change the value of a parameter, uncomment the parameter and set the new value.

Parameter

Description

Default

rootDirectory

A REX root directory contains language models and necessary configuration files.

${rex-root}

rblRootDirectory

The directory containing the RBL root for REX to use.

${rex-root}/rbl-je

rejectGazetteers

Additional gazetter files used to reject entities for the given language.

null

snapToTokenBoundaries

Regular expressions and gazetteers may be configured to match tokens partially independent from token boundaries. If true, reported offsets correspond to token boundaries.

true

rejectRegularExpressionSets

Additional regex files used to reject entities.

null

allowPartialGazetteerMatches

The option to allow partial gazetteer matches. For the purposes of this setting, a partial match is one that does not line up with token boundaries as determined by the internal tokenizer. This only applies to accept gazetteers.

false

kbs

Custom list of Knowledge Bases for the linker, in order of priority

null

maxEntityTokens

The maximum number of tokens allowed in an entity returned by Statistical Entity Extractor. Entity Redactor discards entities from Statistical Entity Extractor with more than this number of tokens.

8

customProcessors

Custom processors to add to annotators.

null

acceptRegularExpressionSets

Additional files used to produce regex entities.

null

calculateConfidence

If true, entity confidence values are calculated. Can be overridden by specifying calculateConfidence in the API call.

false

joinerRuleFiles

File containing additional joiner rules.

null

excludedEntityTypes

Entity types to be excluded from extraction.

null

dataOverlayDirectory

An overlay directory is a directory shaped like the data directory. REX will look for files in both the overlay directory and the root directory, using files from both locations. However, if a file exists in both places (as identified by its path relative to the overlay or root data directory), REX prefers the version in the overlay directory. If REX finds a zero-length file in the overlay directory, it ignores both that file and any corresponding file in the root data directory.

null

confidenceThreshold

The confidence value threshold below which entities extracted by the statistical processor are ignored.

-1.0

resolvePronouns

When true, resolve pronouns to person entities.

false

statisticalModels

Additional files used to produce statistical entities for the given language.

You may pass multiple statistical models. The parameter should be formatted in trios of values specfying language, case-sensitivity and the model file, separated by commas. Case-sensitivity can be automatic, caseInsensitive or caseSensitive. For example, setting two models for case-sensitive English and Japanese might look like : eng,caseSensitive,english-model.bin,jpn,automatic,japanese-model.bin

null

acceptGazetteers

Additional gazetteer files used to produce entities for the given language.

null

linkEntities

The option to link mentions to knowledge base entities with disambiguation model. Enabling this option also enables calculateConfidence.

false

caseSensitivity

The capitalization (aka 'case') used in the input texts. Processing standard documents requires caseSensitive, which is the default. Documents with all-caps, no-caps or headline capitalization may yield higher accuracy if processed with the caseInsensitive value.

Can be automatic, caseSensitive or caseInsensitive

caseSensitive

maxResolvedEntities

The maximum number of entities for in-document coreference resolution (a.k.a. chaining).

2000

calculateSalience

If true, entity chain salience values are calculated. Can be overridden by specifying calculateSalience in the API call.

false

retainSocialMediaSymbols

The option to retain social media symbols ('@' and '#') in normalized output

false

statSalienceMode

An option to calculate entity-chain salience with statistical-based calculation (returns 0 or 1) or simple calculation (returns score between 0 and 1)

true

customProcessorClasses

Register a custom processor class.

null

keepEntitiesInInput

The option to keep existing annotated text entities.

false

redactorPreferLength

The option to prefer length over weights during redaction. If true, the redactor will always choose a longer entity over a shorter one if the two overlap, regardless of their user-defined weights. In this case, if the lengths are the same, then weight is used to disambiguate the entities. If false, the redactor will choose the higher weighted entity when two overlap, regardless of the length of the entity string. In this case, if the weights are the same, then the redactor will choose the longer of the two entities.

true

useDefaultConfidence

The option to assign default confidence value 1.0 to non-statistical entities instead of null.

false

linkingConfidenceThreshold

The confidence value threshold below which linking results by the kbLinker processor are ignored.

-1.0

indocType

An option for document entity resolution (also known as entity chaining). Valid values are: HIGH, STANDARD, STANDARD_MINUS or NULL

STANDARD

Default processors: acceptGazetteer, acceptRegex, rejectGazetteer, rejectRegex, statistical indocCoref, redactor, joiner processors

List the set of active processors for an entity extraction run. All processors are active by default. This method provides a way to turn off selected processors. The order of the processors cannot be changed. Note that turning off redactor can cause overlapping and unsorted entities to be returned.

null

supplementalRegularExpressionPaths

The option to add supplemental regex files, usually for entity types that are excluded by #default. The supplemental regex files are located at data/regex/<lang>/accept/supplemental and are not used unless specified.

null

structuredRegionsProcessingType

Configures how structured regions will be processed. It has three values: none, nerModel, and nameClassifier.

none

regexCurrencySplit

Determines if money values should be extracted as MONEY or CURRENCY_AMT and CURRENCY_TYPE. If true, REX tries to extract CURRENCY instead of MONEY.

false

Overlay Data Directory

If your project has a set of unique data files that you would like to keep separate from other data files, you can put them in their own directory, also known as an overlay directory. This is an additional data directory, which takes priority over the default REX data directory.

The overlay directory must have the same directory tree as the provided data directory. If an overlay directory is set, REX searches both it and the default data directory.

  • If a file exists in both places, the version in the overlay directory is used.

  • If there is an empty file in the overlay directory, REX will ignore the corresponding file in the default data directory.

  • If there is no file in the overlay directory, REX will use the file in the default directory.

To specify the overlay directory use:

  1. Create an overlay directory:

    <install-directory>/my-data

  2. Add the overlay directory to the rex-factory-config.yaml file:

    dataOverlayDirectory:
  <install-directory>/my-data

Example 5. Turn Off a Specific Language Gazetteer

  1. Create an overlay directory:

  2. Add an empty file (gaz-LE.bin ) to the overlay directory:

    my-data/gazetteer/eng/accept/gaz-LE.bin

  3. Add the overlay directory to the rex-factory-config.yaml file:

    dataOverlayDirectory:
  <install-directory>/my-data

The default English gazetteer will not be used in calls.


Example 6. Use a Custom German Reject Gazetteer

In the above example, add a reject gazetteer file:

my-data/gazetter/deu/reject/reject-names.txt

Default configuration

The default configuration of the /entities endpoint uses the same default values as the REX Java SDK. It is optimized to be more performance-oriented, with fewer options enabled, than the configuration of Rosette Cloud. Rosette Cloud is configured to provide a fully-functional demonstration environment. [1]

Table 7. Server and Cloud Default Parameters

Feature

Rosette Server

Parameter Setting in Rosette Server

Rosette Cloud

Entity Linking

false (disabled)

linkEntities:false

true (enabled)

Regular Expression Files

no files are loaded

supplementalRegularExpressionPaths null

all files are loaded

Pronominal Resolution

false (disabled)

resolvePronouns: false

true (enabled)

Case Sensitivity

case-sensitive

caseSensitivity: caseSensitive

automatic

Structured Regions

none

structuredRegionsProcessingType: none

nerModel

Entity linking

Text that refers to an entity is called an entity mention, such as “Bill Clinton” and “William Jefferson Clinton”. Rosette connects these two entity mentions with entity linking, since they refer to the same real-world PERSON entity. Linking helps establish the identity of the entity by disambiguating common names and matching a variety of names, such as nicknames and formal titles, with an entity ID.

Rosette uses the Wikidata knowledge base as a base to link Person, Location, and Organization entities. If the entity exists in Wikidata, then Rosette returns the Wikidata QID, such as Q1 for the Universe. If Rosette cannot link the entity, then it creates a placeholder temporary (“T”) entity ID to link mentions of the same entity in the document. However, the TID may be different across documents for the same entity.

Rosette supports linking to other knowledge bases, specifically the DBpedia ontology and the Thomson Reuters PermID. You can also link to one or more custom knowledge bases. Contact BasisTech Support for more information on using the Field Training Kit (FTK) to add your own linking knowledge base.

Entity linking in Rosette Server is off by default, to improve call speed. When entity linking is turned off, Rosette returns the entities with a TID.

You can enable entity linking in Rosette Server for a single call or as a system default in your environment.

  • Per-call: add {"options": {"linkEntities": true}} to your call.

  • Default: edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

    #The option to link mentions to knowledge base entities with #disambiguation model.
#Enabling this option also enables calculateConfidence.
linkEntities: true

By default, linking to DBpedia is turned off. To turn it on:

  • Enable entity linking

  • Add {"options": {"includeDBpediaTypes": true}} to the call

The list of knowledge bases can be customized in the rex-factory-config.yaml file with the kbs parameter, which takes a List of Paths to knowledge bases.

kbs:
    - /customKBs/kb1
    - /customKBs/kb2
    - /rosette/server/roots/rex/7.44.1.c62.2/data/flinx/data/kb/basis

Note

Setting the list of knowledge bases completely overwrites the list of knowledge bases the linker uses. If you want the default Wikidata knowledge base to be included, it must be on the list of knowledge bases.

Loading regex files

By default, the supplemental regex files are not loaded when the entity extraction endpoint is loaded. To load either the provided supplemental files or new files, edit the /launcher/config/rosapi/rex-factory-config.yaml file adding the supplementalRegularExpressionPaths statements to the file, as show below.

Tip

The files are named following the pattern: data/regex/<lang>/accept/regexes.xml, where <lang> is either the ISO 639-3 language code for the supported language, or xxx for all or any languages.

#The option to add supplemental regex files, usually for entity types that are excluded by
#default. The supplemental regex files are located at data/regex/<lang>/accept/supplemental and
#are not used unless specified.
supplementalRegularExpressionPaths:
- "${rex-root}/data/regex/eng/accept/supplemental/date-regexes.xml"
- "${rex-root}/data/regex/eng/accept/supplemental/geo-regexes.xml"

Pronominal resolution

Entity extraction can try to resolve pronouns with their antecedent entities. For example, in the sentences:

John Smith lives in Boston. He is originally from New York

pronominal resolution would resolve he with John Smith. By default, pronominal resolution is disabled.

To enable it, edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

#The option to resolve pronouns to person entities.
resolvePronouns: true

Case sensitivity

Case sensitivity refers to the capitalization (aka 'case') used in the input texts. Entity extraction can use case to help identify named entities (such as proper nouns) in documents.

Valid values for caseSensitivity

  • caseSensitive: (default) Case found in standard documents, those in which case follows grammar for the most part.

  • caseInsensitive: Used for documents with all-caps, no-caps, or headline capitalization. These are documents in which capitalization is not a good indicator for named entities.

  • automatic: Rosette detects the case from the input model and chooses an appropriate model to use.

To change the default case sensitivity, edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

#The capitalization (aka 'case') used in the input texts. Processing standard documents
#requires caseSensitive, which is the default. Documents with all-caps, no-caps or headline
#capitalization may yield higher accuracy if processed with the caseInsensitive value.
caseSensitivity: automatic

Related articles

Comments

0 comments

Article is closed for comments.





Powered by Zendesk