Entity Linking

By default, all knowledge bases under the data/flinx/data/kb directory inside the REX installation will automatically be used for linking. Any custom knowledge bases placed in this directory will be loaded each time REX launches.

You can enable dynamic loading, controlling which custom knowledge bases will be loaded with EntityExtractor setKbDirs, which takes a List of Paths to knowledge bases.

The list is in priority order; the match from the highest knowledge base on the list will be returned.

The linker processor can associate entities with types drawn from the DBpedia ontology, which provides over 700 types at up to seven levels of granularity.

Providing both primary and secondary entity types increases the usability of the linker processor’s results for many NLP applications. For example, classifying Pheobe Buffay (QID: Q682396) as PERSON is a necessary first step towards effective pronominal resolution, whereas the secondary type path Agent/FictionalCharacter/SoapCharacter paves the way for identifying the relationship of Pheobe Buffay with Lisa Kudrow (Q179041).

Turning on the includeDBpediaType flag increases the recall of the linker processor’s results. When the flag is enabled, the linker will return both non-named entities like "guitar" (Q6607), type MISC, or named entities of new types, such as "2018 World Cup" (Q170645), type EVENT.

By default, providing DBpedia types is turned off. To enable it; EntityExtractor includeDBpediaTypes (boolean includeDBpediaTypes) must be set to true.

The linker processor can return the Refinitiv PermID for a subset of entities which are identified with a QID. By default, linking to PermIDs is turned off.

Entity linking to PermIDs is enabled by setting the EntityExtractor includePermID(boolean includePermID) to true, and disabled by setting it to false. In order to activate PermID linking, both EntityExtractor includePermID(boolean includePermID) and EntityExtractor linkEntities(boolean lnk) must be set to true. When PermID linking isn't explicitly set, its default value is false.

How does entity linking work?

The linker processor first identifies possible entity mentions (“candidates”) through exact matching. It walks through the input text token by token, looking for matches in an aliases.bin file, which contains the alias phrases in the json input files. Then each candidate mention is resolved to a known entity with an ID or labeled as “NONE". NONE here means the candidate mention could not be linked to an entry in the knowledge base.

The disambiguation phase applies a machine-learned SVM ranking model on each candidate mention in isolation. That is, the linking of one candidate doesn’t affect the linking of another. Features for each mention can look at context, but the model is still applied separately to each mention. Context is provided by contextWords and relatedEntityIds.

Generating binaries

In this phase we read and process the knowledge base’s json files, generating easy-to-load binary files. The binary files are used both for candidate generation and selecting features for the disambiguation model.

Different knowledge bases may provide different details about the entities. For example, most knowledge bases can provide information about the prevalence score of each entity, but very few knowledge bases indicate if a phrase is likely to be an entity or not. Those details are used for disambiguation, so a byproduct of this phase is the config file, which lists the available features for the training process.

Training the disambiguation model

After producing a list of the available features, the actual training is performed on a pre-annotated corpus provided by BasisTech for training the disambiguation model. The first step of the process extracts all available features for each annotated entity sample. While there are over a dozen features that could be used, not all may be usable if the required data is not available in the user’s knowledge base. The second step trains the SVM model.

At the end of this phase, both the binaries and the new model are found in the /asset/custom-kb directory, ready to be copied to <rex-home>/data/flinx/data/kb/.

Custom knowledge bases without a disambiguation model

Passing -d as an argument to train-linker-model creates a custom knowledge base without a disambiguation model. Such knowledge bases act as "enhanced gazetteers"; like a gazetteer they will always extract any aliases contained in them. They will also add an ID provided by the knowledge base. Entities extracted from a knowledge base without a disambiguation model will have 1.0 as their linking confidence value. If several entities in the knowledge base have the same alias, one will be returned if the alias is encountered in the text, but which is returned is indeterminate. Overlapping aliases will return the longest one.

A custom knowledge base for the linker processor requires the following files:

For example, if the knowledge base contains entities with information for Japanese and Korean languages, the expected input files are: kb.json, jpn-kb.json and kor-kb.json. If a knowledge base contains only a single language, the kb.json file is not required; all data can go into a single file, the language-specific file (e.g. jpn-kb.json).

[
  {
    "entityId": "B1",
    "entityType": "ORGANIZATION",
    "prevalence": 100,
    "relatedEntityIds": [
      "B2"
    ]
  },
  {
    "entityId": "B2",
    "entityType": "PERSON",
    "prevalence": 95,
    "relatedEntityIds": [
      "B1"
    ]
  }
]

[
 {
    "entityId": "B1",
    "entityName": "Basis Technology",
    "entityType": "ORGANIZATION",
    "aliases": [ {
        "phrase": "Basis Technology"
      }, {
        "phrase": "BasisTech"
      }
    ],
    "contextWords": [ "Software",  "NLP", "Rosette", "Text Analytics" ]
  }, {   "entityId": "B2",
    "entityName": "Carl W. Hoffman",
    "entityType": "PERSON",
    "aliases": [ {
        "phrase": "Carl Hoffman"
      }, {
        "phrase": "Carl"
     }],    
    "contextWords": [ "Software",  "NLP", "Rosette", "Text Analytics" ]  
} ]

The FTK converts your knowledge base definition files into the linker input format, generating binaries from the data, and training the disambiguation model to your custom knowledge base.

docker run -it --rm -v <local-asset-dir-full-path>:/asset -v <rex-installation-path>:/basis/rex rex-field-training

REX field training tools (docker) version
Please refer to the legal notices in /basis/ftk/dependencies/ThirdPartyLicenses.txt.
Copyright (c) 2016 Basis Technology Corporation All Rights Reserved.
Support@basistech.com
 http://www.basistech.com

brat instance started on port 8080 in this container. brat data: /asset/bratdata

Available commands:

Statistical model:
  generate-ngram, cluster-ngram, train-model, evaluate

Advanced tools: learning-curve

Linker model:
 generate-linker-binaries, train-linker-model

Binary gazetteer:
 build-binary-gazetteer

Once the disambiguation model for a custom knowledge base has been created, you can add entries without retraining the disambiguation model. The new entries must be similar to the entries that the model was trained on.

You can add a custom Knowledge Base Connector that retrieves linking information from a knowledge base backed by any external data source or code. Knowledge Base Connectors are an advanced customization feature, and should be used with care; unoptimized connector implementations can greatly affect REX's performance.

Custom Knowledge Base Connectors are created by implementing the com.basistech.rosette.flinx.api.internal.KnowledgeBase interface. The interface contains functions that provide information about entities to the linker disambiguation model features, as described in Train the Model. At a minimum, a connector must implement the findCandidates candidates that identifies potential candidates for linking in a document, the lookupEntityType and getLabels functions that return basic information about entities supported by the Knowledge Base, and the getContextVector function that provides a general context vector for entities.