RNI-Elasticsearch is an Elasticsearch plugin for building fuzzy name retrieval and matching applications for persons, locations, and organizations. The plugin was built with Rosette Name Indexer and supports Elasticsearch .
Important
Due to Elasticsearch version enforcement, this version of the plugin will only work with the same version of Elasticsearch. Contact <support@rosette.com> if you require support for a different version of Elasticsearch.
For a list of third-party components that are used in Basis Technology products, see rlpnc/ ThirdPartyLicenses.txt.
October 2020
New
Known Issues
The doc_score, name_score and date_score functions are not currently supported in percolator queries.
Katakana-Latin name matching with the enableSeq2SeqTokenScorer parameter enabled is not currently supported.
Third-party component updates
October 2020
New Features
Now including RNI-RNT 7.32.3.c62.2
-
Addresses are now supported in the pairwise matching endpoint. Matching can be performed between fielded addresses, unfielded addresses, or a combination of the two.
-
Example:
curl -XPOST 'localhost:9200/rni_plugin/_pair_match?type=rni_address' -H "Content-Type: application/json" -d '
{
"dataPair": {
"data1": {
"houseNumber": "101",
"road": "Main st",
"city": "Cambridge",
"state": "MA",
"country": "USA"
},
"data2": "101 Main St, Cambridge, Massachusetts, United States of America"
}
}'
{
"score": 0.9,
"type": "ORIGINAL"
}
Bug Fixes
Third-party component updates
No changes other than supporting RNI 7.32.3.c62.2
June 2020
New Features
Bug Fixes
Fixed a bug where nested queries using the rni_rescorer that did not specify the window_size parameter would cause a NullPointerException (RLPNC-5394)
Fixed a bug where the windowSizeAllowance and scoreToRescoreRestriction parameters would not properly restrict the first-pass results being rescored in a nested query (RLPNC-5393)
Known Issues
Third-party component updates
April 2020
Known Issues
The doc_score, name_score, and date_score functions currently are not supported in percolator queries.
Katakana-Latin name matching with the enableSeq2SeqTokenScorer parameter enabled is not currently supported
Bug fixes
Third-party component updates
No changes
April 2020
New Features
Known Issues
The doc_score, name_score, and date_score functions currently are not supported in percolator queries.
Katakana-Latin name matching with the enableSeq2SeqTokenScorer parameter enabled is not currently supported
Third-party component updates
No changes other than supporting Elasticsearch 7.6.1
March 2020
New Features
Support Elasticsearch 7.5.2 and include RNI 7.30.5.c62.2
Added a new rescore syntax which must be used when using the score_to_rescore_restriction or window_size_allowance parameters (RLPNC-5263)
Added the parameter enableDynamicConfigurationEndpoints to enable endpoints for stopwords, token overrides and parameters (RLPNC-5225)
Known Issues
It is necessary to add elasticsearch\plugins\rni\bt_root\rlp\bin\amd64-w64-msvc120\ to PATH when installing on Windows
The doc_score, name_score and date_score functions currently are not supported in percolator queries.
Katakana-Latin name matching with the enableSeq2SeqTokenScorer parameter enabled is not currently supported
Bug Fixes
Fixed an issue where dynamic parameters were not loaded correctly in clusters with multiple nodes (RLPNC-5226)
It is now possible to define parameters for multiple parameter universes in a single session (RLPNC-5261)
Third-party component updates
No changes other than supporting Elasticsearch 7.5.2 and RNI 7.30.5.c62.2
Archive Release 7.4.2.0 and Earlier
New Features and Bug Fixes
New Features and Bug Fixes in 7.4.2.0
New Features
Support Elasticsearch 7.4.2 and including RNI 7.30.1.c62.0
Added endpoints to add and remove stopwords (RLPNC-5008)
The Explain API now identifies which indexed name matches the query name (RLPNC-5215)
Known Issues
The doc_score, name_score and date_score functions currently are not supported in percolator queries.
Katakana-Latin name matching with the "enableSeq2SeqTokenScorer" parameter enabled is not currently supported
New Features and Bug Fixes in 7.3.2.0
New Features and Bug Fixes in 7.2.1.0
New Features and Bug Fixes in 6.7.2.3
New Features
Includes RNI 7.29.0.c61.0
Added support for the null_value parameter, allowing nulls to be indexed and searched (RLPNC-5076).
NameScoreFunction no longer sorts results (RLPNC-5132).
New Features and Bug Fixes in 6.7.2.2
New Features and Bug Fixes in 6.7.2.1
New Features and Bug Fixes in 6.7.2.0
New Features
Supporting Elasticsearch 6.7.2.
Added endpoints to add, update and delete parameter universes and token overrides. (RLPNC-4929, RLPNC-4987)
New Features and Bug Fixes in 6.7.1.0
New Features
Supporting Elasticsearch 6.7.1.
Added a new config parameter "ignoreMalformed" which allows the RNI plugin to index documents with unsupported languages but search capabilities are still limited to supported languages. (RLPNC-5001)
Bug Fixes
Updated the pairwise match endpoint to include all match result tuple information for name matching. (RLPNC-5063)
Addressed an assertion statement in a score function which overlooked some edge cases and sometimes failed. (RLPNC-5050)
New Features and Bug Fixes in 6.6.2.1
New Features and Bug Fixes in 6.6.2.0
New Features and Bug Fixes in 6.6.0.0
New Features
Added English address matching. (RLPNC-4352, RLPNC-4353, RLPNC-4999)
Supporting Elasticsearch 6.6.0 and including RNI 7.26.1.c60.0.
New Features and Bug Fixes in 6.5.1.0
New Features and Bug Fixes in 6.4.2.0
New Features and Bug Fixes in 6.3.2.0
New Features
Supports Elasticsearch 6.3.2 and includes RNI 7.24.0.c59.3.
Added pairwise match endpoint for all our supported field types. (RLPNC-4777, RLPNC-4806, RLPNC-4807, RLPNC-4837)
Bug Fixes
Fixed a bug that forced object initialization within the plugin to look for RNI-RNT installation. (RLPNC-4852)
Fixed a bug where DocScoreFunction explanation returned invalid json if using score_if_null. (RLPNC-4881)
New Features and Bug Fixes in 6.2.4.0
No changes other than supporting Elasticsearch 6.2.4 and including RNI 7.23.3.c59.2.
New Features and Bug Fixes in 6.2.3.1
No changes other than supporting Elasticsearch 6.2.3 and including RNI 7.23.1.
New Features and Bug Fixes in 6.2.3.0
No changes other than supporting Elasticsearch 6.2.3 and including RNI 7.23.0.
New Features and Bug Fixes in 6.2.2.1
No changes other than supporting Elasticsearch 6.2.2 and including RNI 7.21.1.
New Features and Bug Fixes in 6.2.2.0
No changes other than supporting Elasticsearch 6.2.2 and including RNI 7.21.0.
New Features and Bug Fixes in 5.6.3.2
No changes other than including RNI 7.21.1.
New Features and Bug Fixes in 6.1.1.0
No changes other than supporting Elasticsearch 6.1.1
New Features and Bug Fixes in 5.6.3.1
New Features
Added a new Thai/English statistical model for matching to improve Thai/English name match performance. (RLPNC-4429)
Added a new Thai name segmentation dictionary, improving segmentation and match performance. (RLPNC-4421)
Improved Thai transliteration, benefiting translation and match performance. (RLPNC-4547)
Enhanced the Thai stop word list, providing better stop word removal from Thai names during matching. (RLPNC-4461)
Tuned the finalBias value for Thai/Thai and Thai/English name pairs to ensure scores for these matches are in line with those of other languages supported by RNI. (RLPNC-4445)
Greatly improved Arabic/Arabic match performance by adding an edit distance metric. (RLPNC-4508)
Improved match performance for name pairs in which the names are identical when spaces are removed. (RLPNC-4495)
Added a few new entries to English/English token overrides. (RLPNC-4529)
Modified the names of RNI's internal Lucene fields so that they are simpler and standardized. (RLPNC-2506)
Added the ability to disable support for individual languages. See internal_param_defs.yaml in rlpnc/data/etc for more information on how to use this feature. (RLPNC-4558)
Known Issues
While we have fixed a bug involving nested fields and the doc_score function, due to limitations of Elasticsearch, one still cannot rescore nested text fields with the doc_score function. If you desire to rescore nested text fields, you must set the fields to be stored in your mapping.
The name_score, date_score, and doc_score functions currently are not supported in percolator queries. Percolator queries that utilize rni_name or rni_date fields will work, however.
New Features and Bug Fixes in 5.6.3.0
New Features
Added preview support for Thai in name matching and name translation. (RLPNC-4444, RLPNC-4420, RLPNC-4419, RLPNC-4417, RLPNC-4490, RLPNC-4424, RLPNC-4423, RLPNC-4493, RLPNC-4479, RLPNC-4418)
Changed default behavior of Chinese names during Chinese / English matching so that they are assumed to be of Chinese origin unless otherwise specified. (RLPNC-4375)
Improved Russian / English name matching in that Russian names now include multiple translations. (RLPNC-4496)
Greatly improved Chinese / Japanese organization name language detection. (RLPNC-4477)
Improved name matching to account for more substring matches. (RLPNC-4498)
Changed the way rni_name and rni_date fields are constructed so that they may now take null as a value. (RLPNC-4485)
Bug Fixes
Fixed a bug in which nested fields could not be used in the doc_score function. See the Known Issues section below for more details. (RLPNC-4500)
Fixed a bug in which the doc_score function would crash if a field was specified that was not present in the mapping. (RLPNC-4499)
Fixed a thread contention issue that could slow down large numbers of threads.
Known Issues
While we have fixed a bug involving nested fields and the doc_score function, due to limitations of Elasticsearch, one still cannot rescore nested text fields with the doc_score function. If you desire to rescore nested text fields, you must set the fields to be stored in your mapping.
The name_score, date_score, and doc_score functions currently are not supported in percolator queries. Percolator queries that utilize rni_name or rni_date fields will work, however.
New Features and Bug Fixes in 5.5.1.0
New Features
Upgraded the native libraries of the native Linux-compatible release of RNI. We are now using CentOS 6 to build these libraries, as CentOS 5 has reached EOL. The new BT_BUILD value for the Linux package is amd64-glibc212-gcc44. (RLPNC-4278)
Added a new config parameter, engEngFastMode which improves speed for English-English matching by turning off HMM and simplifying queries. For more information, check the documentation in internal_param_defs.yaml. (RLPNC-4357)
Added static and deprecated attributes for config parameters. A "static" parameter is one that always has the value loaded in the default parameter profile; setting a static parameter to a different value in other profiles has no effect whatsoever. A "deprecated" parameter is one that we are proposing to eliminate; binding its value to anything other than the default results in a warning. (RLPNC-4193)
Improved the efficiency of when the HMM is used in the case of English-English name pairs. (RLPNC-2988)
Known Issues
The name_score, date_score, and doc_score functions currently are not supported in percolator queries. Percolator queries that utilize rni_name or rni_date fields will work, however.
New Features and Bug Fixes in 5.4.1.0
No changes other than supporting Elasticsearch 5.4.1
Known Issues
-
The embedded version of Lucene (6.5.1) in Elasticsearch 5.4.1 contains a bug that affects nested queries with ScoreMode set to MAX. For this reason, we recommend not using these types of nested queries in any version of the plugin compatible with Elasticsearch 5.4.1. This bug has been fixed in a future version of Lucene and should no longer be an issue in future Elasticsearch versions. (
LUCENE-7833)
New Features and Bug Fixes in 5.3.2.0
No changes other than supporting Elasticsearch 5.3.2
New Features and Bug Fixes in 5.3.1.0
No changes other than supporting Elasticsearch 5.3.1
New Features and Bug Fixes in 5.2.2.0
New Features and Bug Fixes in 5.0.2.2
New Features and Bug Fixes in 5.0.2.1
New Features
Explain information for rni_date and rni_name fields now includes more detailed match information. (RLPNC-3930, RLPNC-4255)
Reduced the size of the package through the word embeddings datafiles. (RLPNC-4245)
New Features and Bug Fixes in 5.0.2.0
New Features
Added a new doc-level score function, which allows for scoring across all fields in a document. (RLPNC-4152, RLPNC-4153, RLPNC-4177)
Added new API methods for scoring names and dates. (RLPNC-4139)
Removed the minScoreToCheck parameter for the name score function. (RLPNC-4189)
Made Date scoring classes and methods public for API use. (RLPNC-4218)
-
Enhanced semantic matching of tokens in Organization names through use of word embeddings.
Note: This drastically increases the size of the plugin package. To reduce the size, the embeddings dictionaries in plugins/rni/bt_root/rlpnc/data/tvec/multilingual can be removed as long as the corresponding language pairs in parameter_profiles.yaml have 'useEmbedded' set to false. (RLPNC-4173, RLPNC-4201, RLPNC-4219, RLPNC-4244)
New Features and Bug Fixes in 2.4.1.1
New Features and Bug Fixes in 2.4.1.0
New Features
Added a new query parameter, windowSizeAllowance, which sets the general proportion of names to pass to the rescorer. This is used at query time to determine the number of names to rescore based on the commonality of the query name in the index, allowing for more efficient querying. As a result, generally a higher window size can be used. (RLPNC-4151)
Added a new query parameter, scoreToRescoreRestriction, that acts as a more efficient replacement for minScoreToCheck and improves query speed. The minScoreToCheck parameter has been deprecated. (RLPNC-4165)
Removed the requirement for an installed bt_root for client-side Java code that communicates with the Elasticsearch server via the Java API. (RLPNC-4131)
Bug Fixes
Fixed a failure that occurred when querying for a field that is not in some documents with minScoreToCheck above 0. (RLPNC-4087)
Fixed an issue where a query size of 0 or unspecified prevented aggregations from being returned. (RLPNC-4188)
New Features and Bug Fixes in 2.3.4.0
New Features
Implemented experimental support for fuzzy date search. (RLPNC-3991, RLPNC-3992)
is now released separately from . Versioning has changed to align with the version of Elasticsearch that is supported. (RLPNC-4048)
Added ability to retrieve the version of being used by the plugin. (RLPNC-4040)
rni_name fields themselves can no longer be nested objects. To retain nested behavior they must be contained in a nested object. (RLPNC-4065)
Bug Fixes
Fixed a permission issue that prevented the plugin from loading native libraries. (RLPNC-4043)
Using rni_name fields within a nested object no longer causes a MapperParsingException. (RLPNC-4056)
Calling _verify_installation will now function when the Elasticsearch server is using an HTTP port other than the default (9200). (RLPNC-4082)
Fixed a crash that occurred when querying for a field that does not exist in some documents (RLPNC-4087)
For information on previous work done on RNI-Elasticsearch, see the release notes for (versions 7.14.0 and previous).
For a list of third-party components that are used in Basis Technology products, see rlpnc/ThirdPartyLicenses.txt.
No changes other than supporting Elasticsearch 7.4.2 and RNI 7.30.1.c62.0
No changes other than supporting Elasticsearch 7.3.2 and RNI 7.29.1.c61.0
No changes other than supporting Elasticsearch 7.2.1
No changes other than those brought in from RNI 7.29.0.c61.0
No changes other than those brought in from RNI 7.28.1.c61.0.
Supporting Elasticsearch 6.7.2 and changes brought in from RNI 7.28.0.c61.0.
No changes other than supporting Elasticsearch 6.7.1.
No changes other than supporting Elasticsearch 6.6.2 and those brought in from RNI 7.27.1.c60.0.
No changes other than supporting Elasticsearch 6.6.2 and those brought in from RNI 7.26.1.c60.0.
No changes other than supporting Elasticsearch 6.6.0 and those brought in from RNI 7.26.1.c60.0.
No changes other than supporting Elasticsearch 6.5.1 and those brought in from RNI 7.25.0.c60.0.
No changes other than supporting Elasticsearch 6.4.2 and those brought in from RNI 7.24.2.c59.3.
No changes other than supporting Elasticsearch 6.3.2 and those brought in from RNI 7.24.0.c59.3.
No changes other than supporting Elasticsearch 6.2.4 and those brought in from RNI 7.23.3.c59.2.
No changes other than supporting Elasticsearch 6.2.3 and those brought in from RNI 7.23.1.
No changes other than supporting Elasticsearch 6.2.3 and those brought in from RNI 7.23.0.
No changes other than those brought in from RNI 7.21.1.
Updated:
Apache Lucene Core 6.2.1 (Apache License) from 6.0.1
Jackson Annotations 2.8.1 (Apache License) from 2.7.3
Jackson Core 2.8.1 (Apache License) from 2.7.3
Jackson Databind 2.8.1 (Apache License) from 2.7.3
Jackson Dataformat YAML 2.8.1 (Apache License) from 2.7.3
fastutil 6.6.1 (Apache License) from 6.6.0
slf4j 1.7.21 (MIT) from 1.6.3
Updated:
Apache Lucene Core 6.0.1 (Apache License) from 5.2.1
Jackson Annotations 2.7.3 (Apache License) from 2.6.2
Jackson Core 2.7.3 (Apache License) from 2.6.2
Jackson Databind 2.7.3 (Apache License) from 2.6.2
Jackson Dataformat YAML 2.7.3 (Apache License) from 2.6.2