The plugin includes Elasticsearch REST APIs to customize and tune matching through stop words, token overrides, and parameter universes. These endpoints allow you to add and modify these configuration values, without having to restart Elasticsearch.
Tip
To use any of the configuration REST APIs, the parameter enableDynamicConfigurationEndpoints must be set to true in the parameter_profiles.yaml file in the any: profile. By default, this parameter is set to false. These endpoints should be used for testing and tuning only. When the dynamic configuration endpoints are enabled, they can slow the system down considerably.
Tip
To use dynamic configuration endpoints in an elasticsearch deployment using SSL encryption, the RNI Elasticsearch plugin must be aware of the server's certificate file. To accomplish this, start elasticsearch with:
ES_JAVA_OPTS="-Dbt.ssl.certificate=<path_to_certificate>"
The _stopwords endpoint allows you to ADD, GET and DELETE stop words without restarting the Elasticsearch server. See Stop Patterns and Stop Word Prefixes for more detailed information on stop words.
The following properties are used when creating stop words. The entity_type is optional; all other fields are required when adding stop words through the API.
Table 2. Stop Word Properties
Property |
Required |
Description |
"lang"
|
✓ |
ISO 639-3 code for the language of the stop word(s). |
stopword_type
|
✓ |
Type of stop word(s), either regexes or prefixes |
"entity_type"
|
|
Entity type for which to apply the stop word(s), defaults to "ALL". |
"stop words"
|
✓ |
List of stop words to be added. |
The POST_stopword adds one or more stop words. The entity_type field is optional, but the other fields are all required.
curl -XPOST "http://localhost:9200/rni_plugin/_stopwords" -H 'Content-Type: application/json' -d '{
"lang": "eng",
"stopword_type": "prefixes",
"entity_type": "PERSON",
"stopwords": [
"honorable",
"senior correspondent"
]
}'
The GET _stopwords method returns all stop words for a given language and stop word type. You can search by just language or by language and type.
Returns all prefix stop words for PERSON types in English:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng_PERSON"
Returns all regex stop words for ORGANIZATION types in Japanese:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/regexes_jpn_ORGANIZATION"
Returns all prefix stop words in English with no type specified. By default, this list is empty; data will only be returned if you've populated the file with values:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng"
The DELETE _stopwords method deletes a specified stop word.
curl -XDELETE "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng_PERSON/doctor"
The _overrides endpoint allows you to ADD, GET and DELETE token pair overrides without restarting the Elasticsearch server. See Overriding Token Pair Matches for more detailed information on token pair overrides.
The following properties are used when creating token overrides.
Table 3. Token Overrides Properties
Property |
Required |
Description |
"lang1"
|
✓ |
ISO 639-3 code for the language of the tokens to be overridden. |
"lang2"
|
✓ |
ISO 639-3 code for the language of the token overrides. |
"entity_type"
|
|
Entity type of the list of token override pairs, defaults to "ALL". |
"token_pairs"
|
✓ |
List of token override pairs to be added. |
"token1"
|
✓ |
Token of lang1 in the pair to be overridden. |
"token2"
|
✓ |
Token of lang2 in the pair to be the override. |
"score"
|
✓ |
Raw score of the token pair between 0.0 and 1.0. |
"force"
|
|
Indicates whether to force this score to be exactly that value for the given token pair, defaults to false . |
The POST _overrides adds one or more token overrides. As shown in the table above, entity_type and force are optional, but the other fields are required.
curl -XPOST "http://localhost:9200/rni_plugin/_overrides" -H 'Content-Type: application/json' -d'{
"lang1": "eng",
"lang2": "eng",
"entity_type": "PERSON",
"token_pairs":
[{
"token1": "Abigail",
"token2": "Abbey",
"score": 0.74,
"force": true},
{
"token1": "Aleksander",
"token2": "Alex",
"score": 0.74},
{
"token1": "Alfonso",
"token2": "Alphonse",
"score": 0.74},
{
"token1": "Frederica",
"token2": "Federica",
"score": 0.74}]}'
The POST _overrides/_update method updates the score / force values of a given override pair.
curl -XPOST "http://localhost:9200/rni_plugin/_overrides/_update" -H 'Content-Type: application/json' -d'
{
"lang1": "eng",
"lang2": "eng",
"entity_type": "PERSON",
"token1": "Frederica",
"token2": "Federica",
"score": 0.65,
"force": true
}'
The following syntax is also correct. It requires you specify the language profile (using the respective ISO 639-3 codes) the token override pair belongs to and, optionally, the entity type. If the entity type is not specified, it defaults to ALL .
curl -XPOST http://localhost:9200/rni_plugin/_overrides/tokens_eng_eng_PERSON/_update\
token1=Frederica&token2=Federica&score=0.65&force=true
The GET _overrides method returns the overrides of a given language profile.
curl -XGET "http://localhost:9200/rni_plugin/_overrides/tokens_hun_eng_PERSON"
You can also retrieve the score of a given override pair.
curl -XGET "http://localhost:9200/rni_plugin/_overrides/tokens_hun_eng_PERSON?token1=abigel&token2=abigail"
The DELETE _overrides method deletes a given override pair.
curl -XDELETE "http://localhost:9200/rni_plugin/_overrides/tokens_hun_eng_PERSON/abigel+abigail"
The _parameter_universe endpoint allows you to ADD, GET and DELETE parameters through parameter universes, without restarting the Elasticsearch server. See Parameter Universe for more information on tuning parameters with parameter universes.
The POST _parameter_universe method creates a parameter universe and the parameter profiles within the universe. If you try to add a parameter universe that already exists, it overrides it with the new values. The parameter universe method uses the following syntax:
SomeParameterUniverseName/xxx_yyy where xxx_yyy is the language profile the parameters belong to expressed in ISO 639-3 codes. The parameters field expects a list of parameters for the given profile where the naming of the parameters should match the ones declared in parameter_defs.yaml
curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe" -H 'Content-Type: application/json' -d'
{
"profiles": [
{
"name": "SomeParameterUniverseName/any",
"parameters": {
"translatorResultsToKeep": 4,
"deletionScore": 0.269,
"doQueryTokenOverrides": true,
"fieldDeletionScore": 0.27,
"yearDistanceWeight": 0.2
}
},
{
"name": "SomeParameterUniverseName/eng_eng",
"parameters": {
"HMMUsageThreshold": 0.8,
"stringDistanceThreshold": 0.1,
"useEditDistanceTokenScorer": true,
"finalBias": 2.4,
"reorderPenalty": 0.2
}
}
]
}'
The POST _parameter_universe/_update updates the values of existing parameters. You need to provide the parameter universe name along with the parameters and their values. If a parameter doesn’t exist it is ignored.
curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe/_update" -H 'Content-Type: application/json' -d'{ "profiles": [ {
"name": "SomeParameterUniverseName/any",
"parameters": {
"translatorResultsToKeep": 4
}
},
{
"name": "SomeParameterUniverseName/eng_eng",
"parameters": {
"useEditDistanceTokenScorer": false,
"finalBias": 2.4,
"reorderPenalty": 0.2
}
}
]
}'
You can also use query parameters to update the value of a parameter. The name of the parameter universe is included as a path parameter. You must then specify two query parameters: param and value. param is made up of the language profile followed by the name of the parameter, value contains the value of the parameter.
curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName\
/_update?param=eng_eng.reorderPenalty&value=0.2"
The GET _parameter_universe method retrieves a given parameter universe. The name of the parameter universe is provided as a path parameter:
curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"
If you add the name of the profile and parameter, it returns the value of the parameter.
curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"
The DELETE _parameter_universe method deletes the parameter universe provided as a path parameter.
curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"
If you add the name of the profile and parameter, it deletes the parameter from the parameter universe. It will then use the value in the parameter_defs.yaml file.
curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"