The plugin includes Elasticsearch REST APIs to customize and tune matching through stop words, token overrides, and parameter universes. These endpoints allow you to add and modify these configuration values, without having to restart Elasticsearch.
Tip
To use any of the configuration REST APIs, the parameter enableDynamicConfigurationEndpoints must be set to true in the parameter_profiles.yaml file in the any: profile. By default, this parameter is set to false. These endpoints should be used for testing and tuning only. When the dynamic configuration endpoints are enabled, they can slow the system down considerably.
Tip
To use dynamic configuration endpoints in an elasticsearch deployment using SSL encryption, the RNI Elasticsearch plugin must be aware of the server's certificate file. To accomplish this, start elasticsearch with:
ES_JAVA_OPTS="-Dbt.ssl.certificate=<path_to_certificate>"
The _stopwords endpoint allows you to ADD, GET and DELETE stop words without restarting the Elasticsearch server. See Stop Patterns and Stop Word Prefixes for more detailed information on stop words.
The following properties are used when creating stop words. The entity_type is optional; all other fields are required when adding stop words through the API.
Table 6. Stop Word Properties
|
Property
|
Required
|
Description
|
|
lang
|
✓
|
ISO 639-3 code for the language of the stop word(s).
|
|
stopword_type
|
✓
|
Type of stop word(s), either regexes or prefixes
|
|
entity_type
|
|
Entity type for which to apply the stop word(s), defaults to ALL.
|
|
stop words
|
✓
|
List of stop words to be added.
|
The stop word index must exist before you can start adding stop words. To create the index:
curl -s -XPOST "localhost:9200/rni_plugin/_stopwords/_create"
The POST_stopword adds one or more stop words. The entity_type field is optional, but the other fields are all required.
curl -XPOST "http://localhost:9200/rni_plugin/_stopwords" -H 'Content-Type: application/json' -d '{
"lang": "eng",
"stopword_type": "prefixes",
"entity_type": "PERSON",
"stopwords": [
"honorable",
"senior correspondent"
]
}'
The GET _stopwords method returns all stop words for a given language and stop word type. You can search by just language or by language and type.
When no entity type is specified, the stop word is applied to all names in the language, those with and without entity types. Therefore, calls that specify a type such as PERSON or ORGANIZATION will also return all stop words that don't have an entity type specified.
Returns all prefix stop words for PERSON types in English:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng_PERSON"
Returns all regex stop words for ORGANIZATION types in Spanish:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/regexes_spa_ORGANIZATION"
Returns all prefix stop words in English with no type specified. By default, this list is empty; data will only be returned if you've populated the file with values:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng"
The DELETE _stopwords method deletes a specified stop word.
curl -XDELETE "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng_PERSON/doctor"
The _overrides endpoint allows you to ADD, GET and DELETE token pair overrides without restarting the Elasticsearch server. See Overriding Token Pair Matches for more detailed information on token pair overrides.
The following properties are used when creating token overrides.
Table 7. Token Overrides Properties
|
Property
|
Required
|
Description
|
|
lang1
|
✓
|
ISO 639-3 code for the language of the first name in the override pair.
|
|
lang2
|
✓
|
ISO 639-3 code for the language of the second name in the override pair.
|
|
entity_type
|
|
Entity type of the list of token override pairs, defaults to "ALL".
|
|
token_pairs
|
✓
|
List of token override pairs to be added.
|
|
token1
|
✓
|
Tokens of the first name in the override pair; they should be of lang1
|
|
token2
|
✓
|
Token of the second name in the override pair; they should be of lang2.
|
|
score
|
|
Raw score of the token pair between 0.0 and 1.0. If omitted, the value from the nicknameOverrideScore parameter is used.
|
|
force
|
|
Indicates whether to force this score to be exactly that value for the given token pair, defaults to false .
|
The override index must exist before you can start adding token overrides. To create the index:
curl -s -XPOST "localhost:9200/rni_plugin/_overrides/_create"
The POST _overrides adds one or more token overrides. As shown in the table above, entity_type, force, and score are optional, but the other fields are required.
curl -XPOST "http://localhost:9200/rni_plugin/_overrides" -H 'Content-Type: application/json' -d'{
"lang1": "eng",
"lang2": "eng",
"entity_type": "PERSON",
"token_pairs":
[{
"token1": "Abigail",
"token2": "Abbey",
"score": 0.74,
"force": true},
{
"token1": "Aleksander",
"token2": "Alex",
"score": 0.74},
{
"token1": "Alfonso",
"token2": "Alphonse",
"score": 0.74},
{
"token1": "Frederica",
"token2": "Federica",
"score": 0.74}]}'
The GET _overrides method returns the overrides of a given language profile.
curl -XGET "http://localhost:9200/rni_plugin/_overrides/hun_eng_PERSON"
You can also retrieve the score of a given override pair.
curl -XGET "http://localhost:9200/rni_plugin/_overrides/hun_eng_PERSON?token1=abigel&token2=abigail"
The DELETE _overrides method deletes a given override pair.
curl -XDELETE "http://localhost:9200/rni_plugin/_overrides/hun_eng_PERSON/abigel+abigail"
The _parameter_universe endpoint allows you to ADD, GET and DELETE parameters through parameter universes, without restarting the Elasticsearch server. See Parameter Universe for more information on tuning parameters with parameter universes.
The parameter index must exist before you can start adding parameters. To create the index:
curl -s -XPOST "localhost:9200/rni_plugin/_parameter_universe/_create"
The POST _parameter_universe method creates a parameter universe and the parameter profiles within the universe. Use this method to add or update a parameter value in a parameter universe. If you try to add a parameter universe that already exists, it overrides it with the new values. The parameter universe method uses the following syntax:
SomeParameterUniverseName/xxx_yyy where xxx_yyy is the language profile the parameters belong to, expressed in ISO 639-3 codes. The parameters field expects a list of parameters for the given profile, where the naming of the parameters should match the ones declared in parameter_defs.yaml.
curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe" -H 'Content-Type: application/json' -d'
{
"profiles": [
{
"name": "SomeParameterUniverseName/any",
"parameters": {
"translatorResultsToKeep": 4,
"deletionScore": 0.269,
"doQueryTokenOverrides": true,
"fieldDeletionScore": 0.27,
"yearDistanceWeight": 0.2
}
},
{
"name": "SomeParameterUniverseName/eng_eng",
"parameters": {
"HMMUsageThreshold": 0.8,
"stringDistanceThreshold": 0.1,
"useEditDistanceTokenScorer": true,
"finalBias": 2.4,
"reorderPenalty": 0.2
}
}
]
}'
Also supported is a simplified request, where the parameter universe name is included in the URL and the parameters are specified with url parameters.
curl -s -XPOST "localhost:9200/rni_plugin/_parameter_universe/SomeOtherUniverseName?param=eng_eng.finalBias&value=4.0"
The GET _parameter_universe method retrieves parameter universes.
To retrieve all parameter universes in a single call:
curl -XGET "localhost:9200/rni_plugin/_parameter_universe"
To retrieve a given parameter universe, the name of the parameter universe is provided as a path parameter:
curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"
To retrieve all parameters for a specific profile within a parameter universe, add the profile:
curl -XGET "localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng"
If you include the name of the profile and a parameter, it returns the value of the parameter:
curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"
The DELETE _parameter_universe method deletes the parameter universe provided as a path parameter:
curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"
If you add the name of the profile and parameter, it deletes the parameter from the parameter universe. It will then use the value in the parameter_defs.yaml file:
curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"
To delete all parameter universes in the index:
curl -XDELETE "localhost:9200/rni_plugin/_parameter_universe"
To delete all parameters for a specific profile within a parameter universe:
curl -XDELETE "localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng"