The plugin includes Elasticsearch REST APIs to customize and tune matching through stop words, token overrides, and parameter universes. These endpoints allow you to add and modify these configuration values, without having to restart Elasticsearch.
Tip
To use any of the configuration REST APIs, the parameter enableDynamicConfigurationEndpoints
must be set to true
in the parameter_profiles.yaml
file in the any:
profile. By default, this parameter is set to false
. These endpoints should be used for testing and tuning only. When the dynamic configuration endpoints are enabled, they can slow the system down considerably.
The _stopwords
endpoint allows you to ADD, GET and DELETE stop words without restarting the Elasticsearch server. See Stop Patterns and Stop word Prefixes for more detailed information on stop words.
The following properties are used when creating stop words. The entity_type
is optional; all other fields are required when adding stop words through the API.
Table 3. Stop Word Properties
Property |
Required |
Description |
"lang"
|
✓ |
ISO 639-3 code for the language of the stop word(s). |
stopword_type
|
✓ |
Type of stop word(s), either regexes or prefixes |
"entity_type"
|
|
Entity type for which to apply the stop word(s), defaults to "ALL". |
"stop words"
|
✓ |
List of stop words to be added. |
The POST_stopword
adds one or more stop words. The entity_type
field is optional, but the other fields are all required.
curl -XPOST "http://localhost:9200/rni_plugin/_stopwords" -H 'Content-Type: application/json' -d '{
"lang": "eng",
"stopword_type": "prefixes",
"entity_type": "PERSON",
"stopwords": [
\"honorable\",
\"senior correspondent\"
]
}'
The GET _stopwords
method returns all stop words for a given language and stop word type. You can search by just language or by language and type.
Returns all prefix stop words for PERSON types in English:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng_PERSON"
Returns all regex stop words for ORGANIZATION types in Japanese:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/regexes_jpn_ORGANIZATION"
Returns all prefix stop words in English with no type specified:
curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng"
The DELETE _stopwords
method deletes a specified stop word.
curl -XDELETE "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng__PERSON/doctor"
The _overrides
endpoint allows you to ADD, GET and DELETE token pair overrides without restarting the Elasticsearch server. See Overriding Token Pair Matches for more detailed information on token pair overrides.
The following properties are used when creating token overrides.
Table 4. Token Overrides Properties
Property |
Required |
Description |
"lang1"
|
✓ |
ISO 639-3 code for the language of the tokens to be overridden. |
"lang2"
|
✓ |
ISO 639-3 code for the language of the token overrides. |
"entity_type"
|
|
Entity type of the list of token override pairs, defaults to "ALL". |
"token_pairs"
|
✓ |
List of token override pairs to be added. |
"token1"
|
✓ |
Token of lang1 in the pair to be overridden. |
"token2"
|
✓ |
Token of lang2 in the pair to be the override. |
"score"
|
✓ |
Raw score of the token pair between 0.0 and 1.0. |
"force"
|
|
Indicates whether to force this score to be exactly that value for the given token pair, defaults to false . |
The POST _overrides
adds one or more token overrides. As shown in the table above, entity_type
and force
are optional, but the other fields are required.
curl -XPOST "http://localhost:9200/rni_plugin/_overrides" -H 'Content-Type: application/json' -d'{
"lang1": "eng",
"lang2": "eng",
"entity_type": "PERSON",
"token_pairs":
[{
"token1": "Abigail",
"token2": "Abbey",
"score": 0.74,
"force": true},
{
"token1": "Aleksander",
"token2": "Alex",
"score": 0.74},
{
"token1": "Alfonso",
"token2": "Alphonse",
"score": 0.74},
{
"token1": "Frederica",
"token2": "Federica",
"score": 0.74}]}'
The POST _overrides/_update
method updates the score
/ force
values of a given override pair.
curl -XPOST "http://localhost:9200/rni_plugin/_overrides/_update" -H 'Content-Type: application/json' -d'
{
"lang1": "eng",
"lang2": "eng",
"entity_type": "PERSON",
"token1": "Frederica",
"token2": "Federica",
"score": 0.65,
"force": true
}'
The following syntax is also correct. It requires you specify the language profile (using the respective ISO 639-3 codes) the token override pair belongs to and, optionally, the entity type. If the entity type is not specified, it defaults to ALL
.
curl -XPOST http://localhost:9200/rni_plugin/_overrides/tokens_eng_eng_PERSON/_update\
token1=Frederica&token2=Federica&score=0.65&force=true
The GET _overrides
method returns the overrides of a given language profile.
curl -XGET "http://localhost:9200/rni_plugin/_overrides/tokens_eng_eng_PERSON"
You can also retrieve the score of a given override pair.
curl -XGET "http://localhost:9200/rni_plugin/_overrides/tokens_eng_eng_PERSON?token1=Frederica&token2=Federica"
The DELETE _overrides
method deletes a given override pair.
curl -XDELETE "http://localhost:9200/rni_plugin/_overrides/tokens_eng_eng_PERSON/Frederica+Federica"
The _parameter_universe
endpoint allows you to ADD, GET and DELETE parameters through parameter universes, without restarting the Elasticsearch server. See Parameter Universe for more information on tuning parameters with parameter universes.
The POST _parameter_universe
method creates a parameter universe and the parameter profiles within the universe. If you try to add a parameter universe that already exists, it overrides it with the new values. The parameter universe method uses the following syntax:
SomeParameterUniverseName/xxx_yyy
where xxx_yyy
is the language profile the parameters belong to expressed in ISO 639-3 codes. The parameters
field expects a list of parameters for the given profile where the naming of the parameters should match the ones declared in parameter_defs.yaml
curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe" -H 'Content-Type: application/json' -d'
{
"profiles": [
{
"name": "SomeParameterUniverseName/any",
"parameters": {
"translatorResultsToKeep": 4,
"deletionScore": 0.269,
"doQueryTokenOverrides": true,
"fieldDeletionScore": 0.27,
"yearDistanceWeight": 0.2
}
},
{
"name": "SomeParameterUniverseName/eng_eng",
"parameters": {
"HMMUsageThreshold": 0.8,
"stringDistanceThreshold": 0.1,
"useEditDistanceTokenScorer": true,
"finalBias": 2.4,
"reorderPenalty": 0.2
}
}
]
}'
The POST _parameter_universe/_update
updates the values of existing parameters. You need to provide the parameter universe name along with the parameters and their values. If a parameter doesn’t exist it is ignored.
curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe/_update" -H 'Content-Type: application/json' -d'{ "profiles": [ {
"name": "SomeParameterUniverseName/any",
"parameters": {
"translatorResultsToKeep": 4
}
},
{
"name": "SomeParameterUniverseName/eng_eng",
"parameters": {
"useEditDistanceTokenScorer": false,
"finalBias": 2.4,
"reorderPenalty": 0.2
}
}
]
}'
You can also use query parameters to update the value of a parameter. The name of the parameter universe is included as a path parameter. You must then specify two query parameters: param
and value
. param
is made up of the language profile followed by the name of the parameter, value
contains the value of the parameter.
curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName\
/_update?param=eng_eng.reorderPenalty&value=0.2"
The GET _parameter_universe
method retrieves a given parameter universe. The name of the parameter universe is provided as a path parameter:
curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"
If you add the name of the profile and parameter, it returns the value of the parameter.
curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"
The DELETE _parameter_universe
method deletes the parameter universe provided as a path parameter.
curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"
If you add the name of the profile and parameter, it deletes the parameter from the parameter universe. It will then use the value in the parameter_defs.yaml
file.
curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"