Entity Extraction in Rosette Enterprise

Text that refers to an entity is called an entity mention, such as “Bill Clinton” and “William Jefferson Clinton”. Rosette connects these two entity mentions with entity linking, since they refer to the same real-world PERSON entity. Linking helps establish the identity of the entity by disambiguating common names and matching a variety of names, such as nicknames and formal titles, with an entity ID.

Rosette uses the Wikidata knowledge base as a base to link Person, Location, and Organization entities. If the entity exists in Wikidata, then Rosette returns the Wikidata QID, such as Q1 for the Universe. If Rosette cannot link the entity, then it creates a placeholder temporary (“T”) entity ID to link mentions of the same entity in the document. However, the TID may be different across documents for the same entity.

Rosette supports linking to other knowledge bases, specifically the DBpedia ontology and the Thomson Reuters PermID. Note that these features are still in LABS and subject to change in the future.

Entity linking in Rosette Enterprise is off by default, to improve call speed. When entity linking is turned off, Rosette returns the entities with a TID.

You can enable entity linking in Rosette Enterprise for a single call or as a system default in your environment.

By default, the regex and gazetteer files are not loaded when the entity extraction endpoint is loaded. To add either default files or new files, edit the /launcher/config/rosapi/rex-factory-config.yaml file adding the supplementalRegularExpressionPaths statements to the file, as show below.

#The option to add supplemental regex files, usually for entity types that are excluded by
#default. The supplemental regex files are located at data/regex/<lang>/accept/supplemental and
#are not used unless specified.
supplementalRegularExpressionPaths:
- "${rex-root}/data/regex/eng/accept/supplemental/date-regexes.xml"
- "${rex-root}/data/regex/eng/accept/supplemental/geo-regexes.xml"

Entity extraction can try to resolve pronouns with their antecedent entities. For example, in the sentences:

John Smith lives in Boston. He is originally from New York

pronominal resolution would resolve he with John Smith. By default, pronominal resolution is disabled.

To enable it, edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

#The option to resolve pronouns to person entities.
resolvePronouns: true

Case sensitivity refers to the capitalization (aka 'case') used in the input texts. Entity extraction can use case to help identify named entities (such as proper nouns) in documents.

Valid values for caseSensitivity

To change the default case sensitivity, edit the /launcher/config/rosapi/rex-factory-config.yaml file as shown below:

#The capitalization (aka 'case') used in the input texts. Processing standard documents
#requires caseSensitive, which is the default. Documents with all-caps, no-caps or headline
#capitalization may yield higher accuracy if processed with the caseInsensitive value.
caseSensitivity: automatic

A regex can be used to extract new entity types that have predictable patterns, by adding a new XML regex file to either the language-specific or generic location. For example, to add a regex for battalion designations, create a regex file such as:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<!DOCTYPE regexps PUBLIC "-//basistech.com//DTD RLP Regular Expression Config 7.1//EN"
   "urn:basistech.com:7.1:rlpregexp.dtd">
<!-- File: military-unit-regex.xml -->
<regexps>
 <!-- Ordinals, as in: 101st -->
 <define lang="eng" name="num_ordinal">\d*(?:1st|2nd|3rd|4th|5th|6th|7th|8th|9th|0th|11th|12th|13th)</define>

 <!-- Battalion labels, partial list, as in: Bn -->
 <define lang="eng" name="label_battalion">(?:bn|battalion|squadron)</define>

 <!-- Regiment labels, partial list, as in: Cavalry [Regiment] -->
 <define lang="eng" name="label_regiment">(?:armor|cavalry|infantry)(?:\s+regiment)?</define>

 <!-- Battalion designation, as in: 2nd Battalion, 1st Infantry -->
 <regexp lang="eng" type="MILITARY:UNIT">(?)${num_ordinal}\s+${label_battalion}
(?:,\s+${num_ordinal}\s+${label_regiment})?</regexp>
</regexps>

Regexes for rejecting entities follow the same format as accept regexes except wildcard (*) is allowed in the entity type. If it is for rejecting German entities, put it in data/gazetteer/deu/reject. If it is for rejecting entities in multiple languages, put it in data/gazetteer/xxx/reject. For example, the following .txt file in data/gazetteer/eng/reject, rejects the PERSON entity named "George Watson" when processing English documents. If you want to create regexes to reject entities, put the file in data/regex/<lang>/reject, or data/regex/xxx/reject for multiple languages. For example, the following .xml file in data/regex/eng/reject rejects "Baltimore" as a LOCATION entity when processing English documents.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE regexps PUBLIC "-//basistech.com//DTD RLP Regular Expression Config 7.1//EN"
                          "urn:basistech.com:7.1:rlpregexp.dtd">
<regexps>
  <regexp lang="eng" type="LOCATION">Baltimore</regexp>
</regexps>

Note: Lookbehind assertions are not supported.

Regexes can improve the extraction when used with a statistical model or a gazetteer. The supplemental regexes are in the data/regex/<lang>/accept/supplemental directory.

In Rosette Enterprise, the supplemental regexes are activated by default. To remove a file, comment out the file entry in the rex-factory-config.yaml file.

The following languages have supplemental regexes:

In this example, we're adding an additional list of entity names to the existing entity type TITLE.

For example, let's add a list of military ranks to the entity type TITLE by creating a gazetteer file such as:

    # File: military-rank-gazetteer.txt 
    TITLE
    private
    private first class
    lance corporal
    corporal
    sergeant
    ... 

Instead of adding entity types you can define a list of entities to reject if they are matched. These are reject gazetteers.

The format of a reject gazetteer is identical to the format of an accept gazetteer except the wildcard (*) is allowed in the entity type. As with accept gazetteers, they are arranged by language. If, for example, it is for rejecting German entities, put it in data/gazetteer/deu/reject. If it is for rejecting entities in multiple languages, put it in data/gazetteer/xxx/reject.

For example, the following .txt file in data/gazetteer/eng/reject, rejects the PERSON entity named "George Watson" when processing English documents.

PERSON
George Watson

A wildcard entity type would match any types. The value "George Watson" would be rejected from all entity types, not just PERSON.

*
George Watson