RNI-ES Release Notes

New Features

• Support Elasticsearch 7.4.2 and including RNI 7.30.1.c62.0

• Added endpoints to add and remove stopwords (RLPNC-5008)

• The Explain API now identifies which indexed name matches the query name (RLPNC-5215)

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

• Katakana-Latin name matching with the "enableSeq2SeqTokenScorer" parameter enabled is not currently supported

New Features

• Support Elasticsearch 7.3.2 and including RNI 7.29.1.c61.0

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

New Features

• Support Elasticsearch 7.2.1

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

New Features

• Includes RNI 7.29.0.c61.0

• Added support for the null_value parameter, allowing nulls to be indexed and searched (RLPNC-5076).

• NameScoreFunction no longer sorts results (RLPNC-5132).

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Bug Fixes

• Implemented a fix to correctly retrieve results for a rescore query after restarting an Elasticsearch instance. (RLPNC-5124)

New Features

• Supporting Elasticsearch 6.7.2 and including RNI 7.28.1.c61.0.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Bug Fixes

• Fixed the redundant reading of RNI config file. Added the parameter "IgnoreBadData", which can be found in bt_root/rlpnc/data/etc/parameter_profiles.yaml file. The "ignoreBadData" parameter can be used to index unsupported language names.

New Features

• Supporting Elasticsearch 6.7.2.

• Added endpoints to add, update and delete parameter universes and token overrides. (RLPNC-4929, RLPNC-4987)

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Bug Fixes

• Fixed the date score function to be used with a valid RNI License only. (RLPNC-5019)

New Features

• Supporting Elasticsearch 6.7.1.

• Added a new config parameter "ignoreMalformed" which allows the RNI plugin to index documents with unsupported languages but search capabilities are still limited to supported languages. (RLPNC-5001)

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Bug Fixes

• Updated the pairwise match endpoint to include all match result tuple information for name matching. (RLPNC-5063)

• Addressed an assertion statement in a score function which overlooked some edge cases and sometimes failed. (RLPNC-5050)

New Features

• Supporting Elasticsearch 6.6.2 and including RNI 7.27.1.c60.0.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

New Features

• Supporting Elasticsearch 6.6.2 and including RNI 7.26.1.c60.0.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

New Features

• Added English address matching. (RLPNC-4352, RLPNC-4353, RLPNC-4999)

• Supporting Elasticsearch 6.6.0 and including RNI 7.26.1.c60.0.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

New Features

• No changes other than supporting Elasticsearch 6.5.1 and including RNI 7.25.0.c60.0.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

New Features

• Supports Elasticsearch 6.4.2 and includes RNI 7.24.2.c59.3.

• Improved documentation on override mechanism. (RLPNC-4866)

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

New Features

• Supports Elasticsearch 6.3.2 and includes RNI 7.24.0.c59.3.

• Added pairwise match endpoint for all our supported field types. (RLPNC-4777, RLPNC-4806, RLPNC-4807, RLPNC-4837)

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Bug Fixes

• Fixed a bug that forced object initialization within the plugin to look for RNI-RNT installation. (RLPNC-4852)

• Fixed a bug where DocScoreFunction explanation returned invalid json if using score_if_null. (RLPNC-4881)

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Bug Fixes

• Removed the log4j properties file so that it doesn't override any existing log4j configuration. (RLPNC-4709)

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Known Issues

• The doc_score, name_score and date_score functions currently are not supported in percolator queries.

Known Issues

• The doc_score function currently is not supported in percolator queries.

New Features

• Added a new Thai/English statistical model for matching to improve Thai/English name match performance. (RLPNC-4429)

• Added a new Thai name segmentation dictionary, improving segmentation and match performance. (RLPNC-4421)

• Improved Thai transliteration, benefiting translation and match performance. (RLPNC-4547)

• Enhanced the Thai stop word list, providing better stop word removal from Thai names during matching. (RLPNC-4461)

• Tuned the finalBias value for Thai/Thai and Thai/English name pairs to ensure scores for these matches are in line with those of other languages supported by RNI. (RLPNC-4445)

• Greatly improved Arabic/Arabic match performance by adding an edit distance metric. (RLPNC-4508)

• Improved match performance for name pairs in which the names are identical when spaces are removed. (RLPNC-4495)

• Added a few new entries to English/English token overrides. (RLPNC-4529)

• Modified the names of RNI's internal Lucene fields so that they are simpler and standardized. (RLPNC-2506)

• Added the ability to disable support for individual languages. See internal_param_defs.yaml in rlpnc/data/etc for more information on how to use this feature. (RLPNC-4558)

Known Issues

• While we have fixed a bug involving nested fields and the doc_score function, due to limitations of Elasticsearch, one still cannot rescore nested text fields with the doc_score function. If you desire to rescore nested text fields, you must set the fields to be stored in your mapping.

• The name_score, date_score, and doc_score functions currently are not supported in percolator queries. Percolator queries that utilize rni_name or rni_date fields will work, however.

New Features

• Added preview support for Thai in name matching and name translation. (RLPNC-4444, RLPNC-4420, RLPNC-4419, RLPNC-4417, RLPNC-4490, RLPNC-4424, RLPNC-4423, RLPNC-4493, RLPNC-4479, RLPNC-4418)

• Changed default behavior of Chinese names during Chinese / English matching so that they are assumed to be of Chinese origin unless otherwise specified. (RLPNC-4375)

• Improved Russian / English name matching in that Russian names now include multiple translations. (RLPNC-4496)

• Greatly improved Chinese / Japanese organization name language detection. (RLPNC-4477)

• Improved name matching to account for more substring matches. (RLPNC-4498)

• Changed the way rni_name and rni_date fields are constructed so that they may now take null as a value. (RLPNC-4485)

Bug Fixes

• Fixed a bug in which nested fields could not be used in the doc_score function. See the Known Issues section below for more details. (RLPNC-4500)

• Fixed a bug in which the doc_score function would crash if a field was specified that was not present in the mapping. (RLPNC-4499)

• Fixed a thread contention issue that could slow down large numbers of threads.

Known Issues

• While we have fixed a bug involving nested fields and the doc_score function, due to limitations of Elasticsearch, one still cannot rescore nested text fields with the doc_score function. If you desire to rescore nested text fields, you must set the fields to be stored in your mapping.

• The name_score, date_score, and doc_score functions currently are not supported in percolator queries. Percolator queries that utilize rni_name or rni_date fields will work, however.

New Features

• Upgraded the native libraries of the native Linux-compatible release of RNI. We are now using CentOS 6 to build these libraries, as CentOS 5 has reached EOL. The new BT_BUILD value for the Linux package is amd64-glibc212-gcc44. (RLPNC-4278)

• Added a new config parameter, engEngFastMode which improves speed for English-English matching by turning off HMM and simplifying queries. For more information, check the documentation in internal_param_defs.yaml. (RLPNC-4357)

• Added static and deprecated attributes for config parameters. A "static" parameter is one that always has the value loaded in the default parameter profile; setting a static parameter to a different value in other profiles has no effect whatsoever. A "deprecated" parameter is one that we are proposing to eliminate; binding its value to anything other than the default results in a warning. (RLPNC-4193)

• Improved the efficiency of when the HMM is used in the case of English-English name pairs. (RLPNC-2988)

Bug Fixes

• Fixed a bug preventing nested rni_date fields from being queryable. (RLPNC-4465)

Known Issues

• The name_score, date_score, and doc_score functions currently are not supported in percolator queries. Percolator queries that utilize rni_name or rni_date fields will work, however.

Bug Fixes

• The parameter "enablePromisingTermFiltering" now defaults to false. This will prevent any issues with indexes that have multiple mappings. Queries can be made faster in Elasticsearch installations with only a single index with only a single mapping by setting this parameter to true.

Known Issues

• The embedded version of Lucene (6.5.1) in Elasticsearch 5.4.1 contains a bug that affects nested queries with ScoreMode set to MAX. For this reason, we recommend not using these types of nested queries in any version of the plugin compatible with Elasticsearch 5.4.1. This bug has been fixed in a future version of Lucene and should no longer be an issue in future Elasticsearch versions. (

  LUCENE-7833

  )

New Features

• Added the ability to choose a score to be returned for a null field in the doc-level score function. (RLPNC-4296)

Bug Fixes

New Features

• Added detailed match information in the explain output for the doc-level score function. (RLPNC-4256)

Bug Fixes

• Fixed an issue that caused errors when the plugin attempted to use word embeddings. (RLPNC-4295)

New Features

• Explain information for rni_date and rni_name fields now includes more detailed match information. (RLPNC-3930, RLPNC-4255)

• Reduced the size of the package through the word embeddings datafiles. (RLPNC-4245)

Bug Fixes

• Fixed an issue that caused errors when trying to load an index containing rni_name fields in Kibana. (RLPNC-4260)

New Features

• Added a new doc-level score function, which allows for scoring across all fields in a document. (RLPNC-4152, RLPNC-4153, RLPNC-4177)

• Added new API methods for scoring names and dates. (RLPNC-4139)

• Removed the minScoreToCheck parameter for the name score function. (RLPNC-4189)

• Made Date scoring classes and methods public for API use. (RLPNC-4218)

• Enhanced semantic matching of tokens in Organization names through use of word embeddings.

  Note: This drastically increases the size of the plugin package. To reduce the size, the embeddings dictionaries in plugins/rni/bt_root/rlpnc/data/tvec/multilingual can be removed as long as the corresponding language pairs in parameter_profiles.yaml have 'useEmbedded' set to false. (RLPNC-4173, RLPNC-4201, RLPNC-4219, RLPNC-4244)

Bug Fixes

Bug Fixes

• Fixed an issue where some documents would fail to be rescored when using query parameters (minScoreToCheck, windowSizeAllowance, and scoreToRescoreRestriction). (RLPNC-4195)

New Features

• Added a new query parameter, windowSizeAllowance, which sets the general proportion of names to pass to the rescorer. This is used at query time to determine the number of names to rescore based on the commonality of the query name in the index, allowing for more efficient querying. As a result, generally a higher window size can be used. (RLPNC-4151)

• Added a new query parameter, scoreToRescoreRestriction, that acts as a more efficient replacement for minScoreToCheck and improves query speed. The minScoreToCheck parameter has been deprecated. (RLPNC-4165)

• Removed the requirement for an installed bt_root for client-side Java code that communicates with the Elasticsearch server via the Java API. (RLPNC-4131)

Bug Fixes

• Fixed a failure that occurred when querying for a field that is not in some documents with minScoreToCheck above 0. (RLPNC-4087)

• Fixed an issue where a query size of 0 or unspecified prevented aggregations from being returned. (RLPNC-4188)

New Features

• Implemented experimental support for fuzzy date search. (RLPNC-3991, RLPNC-3992)

• is now released separately from . Versioning has changed to align with the version of Elasticsearch that is supported. (RLPNC-4048)

• Added ability to retrieve the version of being used by the plugin. (RLPNC-4040)

• rni_name fields themselves can no longer be nested objects. To retain nested behavior they must be contained in a nested object. (RLPNC-4065)

Bug Fixes

• Fixed a permission issue that prevented the plugin from loading native libraries. (RLPNC-4043)

• Using rni_name fields within a nested object no longer causes a MapperParsingException. (RLPNC-4056)

• Calling _verify_installation will now function when the Elasticsearch server is using an HTTP port other than the default (9200). (RLPNC-4082)

• Fixed a crash that occurred when querying for a field that does not exist in some documents (RLPNC-4087)

Added:

• Elasticsearch Rest Client 6.7.2 (Apache License)

• Elasticsearch Rest High Level Client 6.7.2 (Apache License)

Updated:

• MongoDB Java Driver 3.5.0 (Apache License) from 3.4.2

Added:

• MongoDB Java Driver 3.4.2 (Apache License)

Updated:

• Apache Log4j Core 2.9.1 (Apache License) from 2.8.2

• International Components for Unicode (ICU) ICU4J 59.1 (ICU License) from 55.1

Deleted:

• Joda time 2.9.3 (Apache License)

Updated:

• Apache Log4j Core 2.8.2 (Apache License) from 2.7

Updated:

• Apache Log4j Core 2.7 (Apache License) from 2.6.2

Added:

• Apache Log4j Core 2.6.2 (Apache License)

• Elasticsearch Transport Client 5.0.2 (Apache License)

Updated:

• Apache Lucene Core 6.2.1 (Apache License) from 6.0.1

• Jackson Annotations 2.8.1 (Apache License) from 2.7.3

• Jackson Core 2.8.1 (Apache License) from 2.7.3

• Jackson Databind 2.8.1 (Apache License) from 2.7.3

• Jackson Dataformat YAML 2.8.1 (Apache License) from 2.7.3

• fastutil 6.6.1 (Apache License) from 6.6.0

• slf4j 1.7.21 (MIT) from 1.6.3

Added:

• Joda time 2.9.3 (Apache License)

Updated:

• Apache Lucene Core 6.0.1 (Apache License) from 5.2.1

• Jackson Annotations 2.7.3 (Apache License) from 2.6.2

• Jackson Core 2.7.3 (Apache License) from 2.6.2

• Jackson Databind 2.7.3 (Apache License) from 2.6.2

• Jackson Dataformat YAML 2.7.3 (Apache License) from 2.6.2