Rosette Server Release Notes

Known issue: When performing address matching, the setting allLanguageSupport must be set to true. If it is set to false, an unsupported language exception will be thrown, regardless of the language of the address. By default, it is set to true.

Bug fix: The parameter regexCurrencySplit has been fixed. When set to true, currency values will now extract into two entity types: IDENTIFIER:CURRENCY_AMT and IDENTIFIER:CURRENCY_TYPE instead of IDENTIFIER:MONEY. (TEJ-1960)

Improved organization name matching The stopword list for organization entities has been expanded in the following languages (RLPNC-7025):

New embeddings: We've replaced the embeddings used by the /semantics/similar and /semantics/vector endpoints. The new embeddings (GEN_2) provide more accurate results and the results are debiased compared to the previous embeddings. You may see some differences in returned values. The new embeddings are the default. French and Italian continue to use the original GEN_1 embeddings. To use the previous embeddings, set embeddingsMode to GEN_1. (RD-2575)

Bug fix: The deprecated genre parameter is now allowed in document requests.

We've improved the time it takes to tokenize extremely long (> 10K characters) Japanese sentences. (ETROG-3602)

Turkish support added: Added support for Turkish-English and Turkish-Turkish name matching. We have also added person and organization overrides, stopwords, and language detection to improve matching in Turkish. (RLPNC-6499)

Improved person name matching: RNI-RNT now has the ability to detect given names and surnames in Latin script when the name is of English origin. When the enableAdditionalOnomastics parameter is true, gender mismatch penalty is only applied to the detected given name, as opposed to the first name token in a query. (RLPNC-6719, RLPNC-6720)

Improved Arabic person name matching: The new TRAILING_PATRONYMIC_DELETION match phenomenon provides improved scores for matches which contain a deletion that is caused by truncation of a patronymic. The score of this deletion is controlled by the trailingPatronymicDeletionScore parameter. This only applies to Latin script names of Arabic origin when enableAdditionalOnomastics is true. (RLPNC-6756)

Russian support migrated from C++ to Java: Translations from Russian to English may differ slightly. (RLPNC-6764)

We've improved the time it takes to tokenize extremely long (> 10K characters) Japanese sentences. (ETROG-3602)

Bug fix: We fixed a linking problem introduced in 1.24.0, in which the NIL_BIAS parameter value was incorrect. The parameter is now correct for all languages. (TEJ-1899)

Improved Chinese address matching: We've expanded the list of Chinese stop words for addresses. (RLPNC-6587)

Wikidata refreshed: We've updated the knowledge base data for the provided linking knowledge base. The QID assigned to some extracted entities may differ from previous versions. (RWIki-119, ELK-274, ELK-276)

New currency regex: We've introduced a new option, regexCurrencySplit, that, when set to true, will attempt to split entities extracted with the regex engine of type IDENTIFIER:MONEY into two new entities: IDENTIFIER:CURRENCY_AMT and IDENTIFIER:CURRENCY_TYPE. These two new types represent the amount of the currency (50,000) and the currency type ($), respectively. By default, regexCurrencySplit is set to false. (TEJ-1792)

Tagalog support: We've added case-insensitive NER support for Tagalog. Previously we released a case-sensitive model and we've now added the case-insensitive model as well. (TEJ-1858)

Parameter removed: We've removed the deprecated genre extraction option. This option was used to turn the linker on which has been, and will still be, available by the linkEntities option. The genre option is no longer available in the rex-factory-config.yaml file, as an option in the call, or in the Rosette API bindings (TEJ-1855).

Ukrainian support added: Tokenization, sentence boundary detection, segmentation user dictionaries, and many-to-one normalization dictionaries are supported for Ukrainian. (ETROG-3594)

Improved part of speech tags: Language-neutral tokens (numbers, symbols, and punctuation) now get part of speech tags in Indonesian, Standard Malay, and Tagalog. (ETROG-3574)

GPU support: Features that use TensorFlow now use a GPU if available. (ETROG-3564)

Emoji support: Emoji 15.0 is now supported. (ETROG-3577)

New option for Katakana: We've added the option joinKatakanaNextToMiddleDot to control whether sequences of Japanese Katakana tokens adjacent to a middle dot should be merged into a single Katakana token. By default, it is true, which matches the behavior in previous versions of /morphology. This option must be enabled in the rbi-factory-config.yaml file. (ETROG-3592)

Bug fix: The Japanese POS tag VN (verbal noun) is now mapped to the UPT-16 POS tag NOUN. It was previously mapped to VERB. (ETROG-3583)

Improved Japanese organization matching: カンパニー (company) added to the Japanese organization stopwords list. (RLPNC-6545)

Improved Chinese organization matching: We've expanded the list of Chinese stop words for organizations. (RLPNC-6615)

Improved name matching results: When no entityType is specified, the type PERSON will be applied. Previously, the type NONE was applied. (RLPNC-6576)

New parameter for token overrides: We've added a new parameter, overrideSelector, to control which overrides will be considered during querying and matching. Override filenames can now specify a “selector” value which will be matched against this parameter. (RLPNC-6561)

Bug fix: Horizontal tabs are now removed as part of normalization in English. (RLPNC-6541)

Bug fix: Control characters are now removed from Arabic names before matching. (RLPNC-6543)

Bug fix Fixed a case where unexpected name inputs could lead to a null pointer exception. (RLPNC-6634)

Bug fix Fixed an issue where name-similarity could return match scores greater than 1. (RLPNC-6595)

Ukrainian support added: Sentence boundary detection is now supported for Ukrainian. (ETROG-3594)

Ukrainian support added: Tokenization is now supported for Ukrainian. (ETROG-3594)

New option for Katakana: We've added the option joinKatakanaNextToMiddleDot to control whether sequences of Japanese Katakana tokens adjacent to a middle dot should be merged into a single Katakana token. By default, it is true, which matches the behavior in previous versions of RBL-JE. (ETROG-3592)

Tagalog (tgl) support: We've added Tagalog to our list of languages. The following processors are supported: gazetteer, regex, statistical NER, linking. (TEJ-1812, TEJ-1822, TEJ-1785, TEJ-1786)

New linking option: We've added a new option for entity linking. When linkMentionMode is set to entities the linker will attempt to link the entities extracted by other processors (regex, gazetters, and the statistical processor) instead of using its own processor to extract entity candidates. Depending on your data, this may provide higher accuracy and speed. (TEJ-1806)

This option has been added to the rex-factory-config.yaml file and is also available as an endpoint option on a per-call basis.

Parameter deprecated: The parameter genre is deprecated and will be removed in the next release.

Bug fix: An exception is no longer emitted when token normalization produces an empty string. (TEJ-1803)

Bug fix: When looking for candidate mentions in text, if there is an overlap between these mentions, the linker now resolves the longest spanning mention before disambiguation. (ELK-277)

Tagalog (tgl) support:

Indonesian (ind) support: RBL now supports lemmatization for Indonesian, which is the standardized form of Malay spoken in Indonesia. (ETROG-3563)

Standard Malay (zsm) support: RBL now supports lemmatization for Standard Malay, the standardized form of Malay spoken in Malaysia. (ETROG-3563)

Bug fix: We fixed a bug in Russian where certain uncommon consonant–vowel sequences in words in the lexicon were incorrectly replaced with more common sequences with different vowel letters. (ETROG-3541)

Example: брошюра

Complete CJK Ext A support: We now have full support of CJK Unified Ideographs Extension A. (RLPNC-6324)

Improved Spanish name matching: We have improved Spanish surname detection. (RLPNC-6294)

Improved Japanese location matching: All prefectures of Japan are now included in the override list. (RLPNC-6326)

Example: 北海道 vs. Hokkaido Prefecture

Bug fix: The string "luiz arlos da silva bueno" is no longer in the Greek, English, and Vietnamese stop word lists. (RLPNC-6358)

Complete CJK Ext A support: We now have full support of CJK Unified Ideographs Extension A. (RLPNC-6324)

Tagalog (tgl) support: The Tagalog sentence-breaker now recognizes certain abbreviations that end with periods and doesn’t break sentences after them. The tokenizer keeps the period in the token with the rest of the abbreviation. (ETROG-3573)

Tagalog (tgl) support: The Tagalog sentence-breaker now recognizes certain abbreviations that end with periods and doesn’t break sentences after them. The tokenizer keeps the period in the token with the rest of the abbreviation. (ETROG-3573)

Use custom linking knowledge base as an entity extraction gazetteer: We've added the parameter disableLinkerDisambiguation, which was previously only available in the REX SDK. This parameter lets you use the knowledge base as a gazetteer. Each exact match is assigned an ID from the knowledge base without running disambiguation. This disables checking that the context of the document is similar to the context of the knowledge base. (ELK-270)

Configure knowledge base linking priority: With multiple knowledge bases it is possible to set the order in which to try linking against each knowledge base. Set the priority in the redactor configuration file (ne_types.xml) (TEJ-1726, TEJ-1754)

Example: The following XML element will set the custom-kb priority higher than the default knowledge base (kb-linker) when linking a PRODUCT entity type:

<ne_type>
  <name>PRODUCT</name>
  <weight name="kb-linker" value="100" />
  <weight name="kb-linker:custom-kb" value="1" />
</ne_type>

relatedEntities renamed to contextWords: When creating a custom knowledge base, the feature contextWords, which was previously called relatedEntities, is required. Context words are language-specific words that are strongly related to the entity. The term relatedEntities has been deprecated. (TEJ-1756)

Bug fix: An error is no longer generated when there are null prefixes in Arabic morphological analyses. (TEJ-1765)

Bug fix: We fixed a bug to enable using noisy_context_vector feature for disambiguation. (ELK-265, ELK-268, ELS-272, TEJ-1776)

Indonesian support added: Rosette now supports part of speech (POS) tagging in Indonesian. (ETROG-3543)

Malay (standard) support added: Rosette now supports part of speech (POS) tagging in Malay (standard). (ETROG-3545)

Russian lexicon improved: We've added many words related to computer technology to the Russian lexicon. (ETROG-3523, ETROG-3538)

Bug fix: In Japanese, negative forms of Ichidan verbs written all in Hiragana are no longer lemmatized to end with “なう”. (ETROG-3534)

Example: Input: くれない

Improved matching of Spanish names and names of Spanish origin: RNI now has a deeper understanding of Spanish surnames. For example: "JOSE JORGE RIOS TORRES" now gets a higher score when matched against "JOSE RIOS" than it does when matched against "JOSE TORRES", since "RIOS" is recognized as the primary surname. (RLPNC-6037)

The following parameters impact how Spanish names are matched:

Improved matching of English organization names: We've added ordinal numbers to the override list for English organizations. (RLPNC-6225)

Example: 1st National Bank vs. First National Bank

Improved Vietnamese name matching: We've expanded the Vietnamese stop word lists for PERSON and ORGANIZATION entity types. (RLPNC-5694)

Example: Chủ tịch Hồ Chí Minh vs. Hồ Chí Minh (translation: President Ho Chi Minh)

New parameter for non-phonetic matches: We've added the parameter editDistanceScoreBias to adjust the bias for edit distance scores. Increasing the impact of edit distance scores can improve the match scores of typographical errors and other non-phonetic matches. (RLPNC-6199)

New parameter for organization names: We've added the parameter tokenizeOrganizationsWithNumbers that prevents tokenization of names with numbers within the name. When set to true (default), the number is left within the token and the name will get a higher value from the edit distance scorer. This is desirable if your data contains organization names which intersperse alphabetic and numeric characters or if your data often contains typographical errors with numerals inserted into otherwise valid tokens. When set to false, the number remains within the organization name token. (RLPNC-6200)

Support for Cantonese name transliterations: We've added Jyutping transliterations (Cantonese pronunciation of Chinese names) to the list of readings. The new parameter enableYueReadings enables Jyutping readings. It is set to false by default. To enable Jyupting readings, set enableYueReadings to true. (RLPNC-6239)

Improved Japanese-English location name matching: We've expanded the Japanese-English overrides list for location names. (RLPNC-6268)

Example: 大阪府 vs. Osaka Prefecture

Bug Fix: Name similarity will no longer return match scores above 1.0. (RLPNC-6254)

Cantonese support added: RNT can now transliterate Han into Latin characters using the Jyutping transliteration scheme for Cantonese. (RLPNC-6232)

Improved Chinese - English address matching: We expanded overrides for ethnic minority regions, particularly from Xinjiang, Tibet, and Inner Mongolia. (RLPNC-6077)

Bug fix: Address matching no longer emits a NullPointerException in certain cases of address matching in which an address contains multiple languages. (RLPNC-6083)

Spaceless Korean tokenizer: We've added an option to select the spaceless Korean tokenizer in the call. The default tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. To use this tokenizer, set the option modelType to DNN. Previously, this option was not available on a per-call basis. (ETROG-3513)

Khmer - English added: Khmer - Khmer and Khmer - English are now supported name matching pairs. (RLPNC-5712)

Improved language detection: We've improved language detection for languages that use Han characters (Chinese, Japanese, Korean). (RLPNC-6059)

Improved ORG matching: We've expanded the list of known organization names in our real world ID tables to improve ORG matching in Arabic (ara), Burmese (mya), Chinese (zho), French (fra), German (deu), Greek (ell), Hebrew (heb), Hungarian (hun), Italian (ita), Japanese (jpn), Korean (kor), Portuguese (por), Russian (rus), Spanish (spa), Thai (tha), and Vietnamese (vie). (RLPNC-6090)

New parameter boostWeightAtLeftEnd: We added a new parameter boostWeightAtLeftEnd to increase the weighting of the first token in a name. When setting this parameter, the boostWeightAtRightEnd parameter should not be modified. (RLPNC-6094)

Improved Chinese - English ORG matching: We added override mappings for Chinese numerals in Hanzi to Arabic numbers from zero through twenty-one. (RLPNC-6028)

Bug fix: Pairwise match now works with all languages that have limited language support. Previously, an error was returned for unidentified languages. (RLPNC-6100)

Khmer - English added: Khmer to English is now a supported name translation language pair. (RLPNC-5708)

New Language support: The Semantic Similarity endpoints now support Tagalog (tgl). (RD-2546)

Spaceless Korean tokenizer: We've added an option to select the spaceless Korean tokenizer in the call. The default tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. To use this tokenizer, set the option modelType to DNN. Previously, this option was not available on a per-call basis. (ETROG-3513)

Improved address searching: The set of address match results returned are now consistent. (RLPNC-6057)

Updated log4j to version 2.17.1 to implement fixes for the vulnerabilities identified in CVE-2021-44832.

Updated log4j to version 2.17 to implement fixes for the vulnerabilities identified in CVE-2021-45105.

Updated log4j to version 2.16 to implement fixes for the vulnerabilities identified in CVE-2021-45046.

Chinese address matching: We now support Chinese-Chinese and Chinese-English address matching. (RLPNC-5822)

Language-specific address override files: Address override files are now language-specific, and the file name must include the language codes. (RLPNC-6032)

Example: English-English state overrides

Example: Chinese-English city overrides

Language-specific address stop word files: Stop word files for address matching on text fields (house, road, city, state, country) are now language-specific, and the file name must include the language code (either eng or zho). (RLPNC-6031)

Example: English city stop pattern

Bug fix: Hungarian dates are now extracted correctly. Previously, dates with embedded periods followed by a space were not being extracted. (TEJ-1681)

Bug fix: The entities/info endpoint now returns the complete list of valid entity types. (WS-2360)

Bug fix: The entities/info endpoint no longer lists TEMPORAL types by default for SWEDISH. (TEJ-1687)

Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts in Japanese. (ETROG-3474)

Example: Input: 三・一四

Emojis: U+3030 and U+303D are now tagged as emojis even when not followed by U+FE0F. (ETROG-3478)

Emoji support: We now support the emoji in Unicode 14.0 (ETROG-3476)

Japanese tokenization: In Japanese, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. (ETROG-3475)

Improved POS tags: Many number, punctuation, and symbol characters are now POS-tagged appropriately as numbers, punctuations, and symbols instead of being marked as unknown or some other tag. This applies to all languages with POS tags. (ETROG-3481)

Hungarian improvements: We've added some Hungarian abbreviations and improved sentence boundary detection around Hungarian abbreviations. (ETROG-3479, ETROG-3484)

Bug fix: In Japanese, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)

Bug fix: We've reverted two of the POS changes made in version 1.19.0 as they introduced regressions in Chinese and Japanese. (ETROG-3466)

The values are now:

Bug fix: Morphology no longer detects characters as emoji when followed by the text presentation selector (U+FE0E). (ETROG-3480)

Bug fix: In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)

Basic support for all languages: Name similarity can now match names in any language. Languages which previously would have returned an "unsupported language" error now return a match score. The score is either 1 for a perfect match, or a value based on edit distance. Set the parameter allLanguageSupport to false for backwards compatible behavior to previous versions. (RLPNC-5979)

Improved ORG matching: We added real world ID tables or organizational names to improve ORG matching in the following languages: Thai (tha), Greek (ell), Hebrew (heb), Burmese (mya), German (deu), French (fra), Hungarian (hun), Italian (ita), Portuguese (por), Spanish (spa), and Vietnamese (vie). (RLPNC-5986)

Example: "International Astronomical Union" vs. "האיגוד האסטרונומי הבינלאומי"

Neural model for Katakana: We've added a neural-based phonetic matching model to improve Katakana-Latin name matching. To enable the model, set enableSeq2SeqTokenScorer to true in internal_param_defs.yaml file. (RLPNC-5945)

NOTE: This feature is not currently available in Rosette Server.

Improved Hebrew name matching: We've added a rule-based vocalization checker for the statistical-model vocalizer to improve Hebrew-Hebrew and Hebrew-English name matching. (RLPNC-5990)

Bug fix: Burmese-English transliteration has been improved by revising the Folk and MLCTS transliteration schemes. (RLPNC-5950)

Bug fix: Hebrew Folk transliteration has been improved, especially for the letters vav and yod. (RLPNC-5916)

Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts in Japanese. (ETROG-3474)

Example: Input: 三・一四

Japanese tokenization: In Japanese, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. (ETROG-3475)

Bug fix: In Japanese, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)

Bug fix: In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)

Improved address matching: We've modified the field weight values to provide more accurate address match scores. Weightings were determined by evaluating US and UK address data. (RLPNC-5893)

Example: "85 Court Road Newton Ferrers, Plymouth PL8 1DE1B Devon, England UK” vs “85 Court Road Newton Ferrers PL8 1DE UK"

Improved address overrides: Address overrides are now applied to groups of related address fields, instead of just individual fields. Overrides apply when matching any two fields from the same group. (RLPNC-5899)

Bug fix: The supported entity types info of the DNN processor now spells PERSON correctly. (TEJ-1670)

ARM64 support: We now support ARM64 processors. (RLPNC-5912)

Burmese transliteration: We added a Basis Technology-created Folk transliteration scheme for Burmese name matching that is similar to how Burmese names are commonly transliterated to English. (RLPNC-5892)

Improved Hebrew transliteration: The Hebrew character ח used to be transliterated as “h” in some cases and “kh” in others (if it was followed by a geresh). It is now transliterated as “ch"when not followed by a geresh. The Hebrew character כ used to be transliterated as “h” in some cases and “k” in others (if it has a dagesh). Now, it is transliterated to “ch” in the cases when it used to be transliterated to “h”. (RLPNC-5928)

Example: נחמן

Example: מיכל

Improved address matching: We've expanded the override tables for UK, U.S., and Canadian addresses. (RLPNC-5886)

Example: houseNumber<47>road<Albert Street>city<Aberdeen>stateDistrict<Aberdeenshire>postCode<AB25 1XT> vs. houseNumber<47>road<Albert Street>city<Aberdeen>stateDistrict<ABD>postCode<AB25 1XT>

Bug fix: Overrides for alphanumeric address fields (houseNumber, unit, poBox, postCode) are now being applied. (RLPNC-5863)

Example: “3710 W Martin Luther King Blvd STE #121” vs. “3710 W Martin Luther King Blvd Suite #121”

Wikidata refreshed: The internal database for Wikidata linking has been refreshed and re-indexed. QIDs for some entities may change from previous versions. (TEJ-1657, TEJ-1658)

New /info endpoint: The entities/info endpoint returns a list of supported entity types available by processor type and language. If you provide a profileId, it returns the list of entity types in the model used by a custom profile. Set the optional parameter perSubsourceSupportedEntityTypes to true to list the entity types by subsource.

New option for structured regions: The option structuredRegionProcessingType has been added to specify the type of processing for structured regions. By default, the statistical/DNN model is turned off for structured regions, which increases precision, but may result in reduced recall in structured regions.

The existing option enableStructuredRegion does not determine how structured regions get processed. When enableStructuredRegion is set to true and the input is contentUri (i.e. set to retrieve content from a URL), then HTML lists and tables will be extracted by Tika as structured regions. The value of structuredRegionProcessingType will then determine how those structured regions are processed.

Bug fix: Confidence scores for language regions are now always within the range [0,1]. (RLIJE-552)

Improved Hebrew-English name matching:

Improved Hebrew-Hebrew name matching: We expanded the token overrides for person entity types. (RLPNC-5891)

Example: סולומונתאס vs. סולונאס

Improved English-English name matching: We added the token override pair Alex/Aleksandar. (RLPNC-5871)

Example: Alex vs. Aleksandar

Improved matching for identifiers: We improved matching and added support for three new subtypes: IDENTIFIER_DRIVERS_LICENSE, IDENTIFIER_LICENSE_PLATE, IDENTIFIER_NATIONAL_ID_NUM, along with IDENTIFIER_GENERIC. (RLPNC-5852)

Example: NH123456789DL vs. NH123456789DN (as IDENTIFIER_DRIVERS_LICENSE entity type)

Improved Japanese Segmentation: We've expanded the segmentation dictionary to improve Japanese name segmentation. (RLPNC-5835)

Example: ミロシェヴィッチスロボダン

Bug fix: Hebrew tokens containing diacritics are now identified in the override table. (RLPNC-5882)

Example: אֲבִי vs. Abigail

New language support: Sentence tagging is now supported for Indonesian, Standard Malay, and Tagalog. (ETROG-3443)

New language support: Tokenization is now supported for Indonesian, Standard Malay, and Tagalog. (ETROG-3443)

When using DNN models in the /entities endpoint, a java.lang.OutOfMemoryError would sometimes occur. A configuration setting has been changed, eliminating the error. This only occurred in the 1.19.1 release.

Improved handling of postal codes: Better match scores result from this enhancement. (RLPNC-5639)

Example: houseNumber<123>road<Clifton St>city<Cambridge>state<MA>postCode<02140 1234> vs. houseNumber<123>road<Clifton St>city<Cambridge>state<MA>postCode<02140-1234>

Expanded UK and CA address override tables: We expanded override tables for UK and Canadian addresses. (RLPNC-5607)

Example: houseNumber<100>road<Main Ave>city<Shellbrook>state<Saskatchewan>postCode<S0J 2E0> vs houseNumber<100>road<Main Ave>city<Shellbrook>state<Sask>postCode<S0J 2E0>

Bug fix: Invalid whitespace handling by the DNN processor no longer causes a runtime exception. (TEJ-1614)

New option for tokenizers: We've added a new option, tokenizerType to specify which tokenizer to use. The options alternativeTokenization and fstTokenize are deprecated in favor of tokenizerType. See Tokenizer Types fore more detail. (ETROG-3419)

New Korean tokenizer: We've added a new tokenizer for spaceless Korean input. The previous tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. Activate it by setting tokenizerType to spaceless_statistical. (ETROG-3392)

Bug fix: Consecutive punctuation characters are no longer returned as a single token in Chinese when alternativeTokenization is true or tokenizerType is set to spaceless_lexical. Now each character is its own token. (ETROG-3402)

Example: Input: 天津？？

Bug fix: Enabling alternativeSpanishDisambiguation for Spanish no longer causes an error. (ETROG-3435)

Bug fix: Setting disambiguatorType to DNN for Hebrew no longer causes an error. (ETROG-3437)

Improved name deduplication: We added support for name overrides so that, for example, the nickname “Mike” is included in the same cluster with “Michael”. (RLPNC-5600)

Burmese-English added: Burmese-Burmese and Burmese-English are now supported name matching pairs. (RLPNC-5660)

Hebrew-English added: Hebrew-Hebrew and Hebrew-English are now supported name matching pairs. (RLPNC-5339)

Vietnamese-English added: Vietnamese-Vietnamese and Vietnamese-English are now supported name matching pairs. (RLPNC-5687)

Improved Chinese-English: Name similarity scores for Chinese-English now leverage English translations from a list of Chinese name translations. (RLPNC-5643)

Example: 汤姆 vs. Tom

Improved English-English organizations: We expanded the overrides list with numbers and their written form from 1 to 21. (RLPNC-5644)

Example: Channel One Russia vs. Channel 1 Russia

Improved name matching for organizations: Name similarity scores for organizations has been improved by adding new frequency models in English, Chinese, Arabic, Japanese, and Russian. (RLPNC-5416)

Updated frequency model: We updated the English frequency model for PERSON entity type by adding birth names from 1920-2019 to the existing model. (RLPNC-5592)

Burmese-English added: Burmese to English is now a supported name translation language pair. (RLPNC-5662)

Example: မင်း အောင် လှိုင် ⟹ Maang Aaung Lhuing

Language-specific joiner rules: Custom joiner rules can now be language-specific or apply to all languages. (TEJ-178)

New default processing for structured text regions (lists, tables): Because structured text is often just words or phrases, and thus missing the syntactic context that REX was trained on, some REX users would pre-process input text to remove structured regions, on which REX performed poorly. Users no longer have to pre-process the input as now the statistical/DNN model is turned off by default for structured regions. This mode increases precision but may result in reduced recall in these regions. Note, the other REX processors (pattern match, exact match, entity linking) which do not rely on context will continue to analyze the structured regions. To turn on the statistical/DNN model for structured regions, set the parameter structuredRegionProcessingType to nerModel. (TEJ-1502) (TEJ-1502)

New name classifier model for structured regions (LABS): We've added a new model for processing structured regions. Each sentence or fragment (the structured region) is classified as a single entity. The name classifier classifies the entity as PERSON, LOCATION, or NONE. It is disabled by default. It can be specified in the rex-factory-config.yaml file or by using the enableStructuredRegion option in the call. (TEJ-1613, TEJ-1621)

New option for structured regions: The option enableStructuredRegion has been added to configure how structured regions are processed. The default value is false. When set to true:

Japanese organization gazetteers: The gazetteers for Japanese organizations has been updated to improve extraction of Japanese organizations. (TEJ-1612)

New RBL version: Entity extraction now consumes the latest version of Rosette Base Linguistics (RBL) 7.39.0. (TEJ-1618)

Rosette Training Server (RTS) results: When using REX with Rosette Adaptation Studio (RAS), the results returned by RTS are now preferred by default. (TEJ-1605)

Bug fix: Entities are no longer extracted when they cross a sentence boundary. To enable entity linking across sentence boundaries, set disableApplySentenceBoundaries to true. (ELK-259)

Bug fix: Entities are now checked to ensure they are normalized. (TEJ-1615)

Bug fix: Changing the value of maxResults no longer changes the results returned in addition to the number of results. Previously, changing the value of maxResults could change the results when there was a small off-script region in the input text. (RLIJE-534)

Improved Hebrew tokenizer and new analyzer: The Hebrew tokenizer is now more consistent with the tokenizers of other languages. (ETROG-3290)

Normalized characters: Normalized half-width and full-width characters are processed the same as their counterparts. (ETROG-3351)

Bug fix: Hebrew tokens containing a geresh are now tokenized properly. Previously, only the part up to the geresh would be returned as the token, and the part after the geresh would sometimes be considered a suffix. Now all the characters of the token are returned as the token. ETROG-3290)

Example: מע'רב

Bug fix: A previous entity offset alignment issue involving very short regex-captured entities followed by \r is fixed.

Bug fix: We reverted some changes to Korean tokenization from the 1.18.0 release. You may see minor differences in extracted entities in Korean when comparing results from the 1.18.0 and 1.18.1 releases.

Greek lexicon: The Greek lexicon has additional nouns.

New Greek disambiguator added: The new Greek disambiguator is more accurate, but slower. The new disambiguator is enabled by default. To use the old disambiguator, set alternativeGreekDisambiguation to true.

Greek disambiguation improved: Certain Greek forms are now disambiguated to prefer a modern analysis over an archaic analysis. alternativeGreekDisambiguation must be set to false, which is the default.

Example: δείξε

Deprecated classes: The classes BufferWordBreaker and WordBreakResults have been deprecated.

Emoji normalization: Morphology no longer normalizes certain emoji zero-width joiner (ZWJ) sequences to U+1F48F KISS, U+1F491 COUPLE WITH HEART, and U+1F46A FAMILY, to be consistent with Unicode’s efforts to make emoji more gender-neutral by default.

Bug fix: The Greek guesser now handles tokens with non-alphanumeric characters.

Example: Start+

Bug fix: We reverted some changes to Korean disambiguation from the 1.18.0 release as the changes introduced new disambiguation errors. You may see minor differences in Korean disambiguation when comparing results from the 1.18.0 and 1.18.1 releases.

Character normalization improved: We've improved character normalization for all supported languages. All languages except for Japanese, Korean, and Arabic-script languages now have NFKC normalizations performed. Japanese, Korean, and Arabic-script languages have NFKD normalizations performed.

Japanese ORGANIZATION type matching improved:  We've improved the accuracy of ORGANIZATION matching for Japanese-English and Japanese-Japanese names by expanding the stop word list for the ORGANIZATION entity type.

Example: コダック合同会社 vs. Kodak Limited

Better entity resolution for Russian ORGANIZATION type matching:  We've improved Russian-English and Russian-Russian name matching by adding support for Russian organizations in the entity resolution engine.

Example: Ура́льские авиали́нии vs. Ural Airlines

Expanded stop word list for Russian ORGANIZATION type matching:  We've improved the accuracy of ORGANIZATION matching for Russian-English and Russian-Russian names by expanding the stop word list for the ORGANIZATION entity type.

Example: Балтийский федеральный университет имени Иммануила Канта vs. Immanuel Kant Baltic Federal University

Korean ORGANIZATION type matching improved:  We've improved Korean-English and Korean-Korean name matching by adding support for Korean organizations in the entity resolution engine.

Example: 현대자동차 vs. Hyundai Motor Company

Bug fix: We fixed a bug where Arabic-Arabic name matching was returning a low score in some cases for names that were seemingly very similar.

Example: عبدلخالق الحوثي vs. عبدالخالق الحوثي

Wikidata refreshed: The internal database for Wikidata linking has been refreshed and re-indexed. QIDs for some entities may change from previous versions.

Bug fix: The sample for the SQLite-kb-connector now works correctly. Runtime issues with SQLite dependencies have been corrected.

Bug fix: Extraction no longer fails when a custom processor returns a NULL annotator; a warning is issued instead.

Bug fix: Mentions normalized by a custom processor are no longer ignored.

Bug fix: Newlines \r and \r\n are now handled correctly.

Deprecated option: The alternative tokenization option deliverExtendedAttributes is now deprecated. Previously it delivered an unsupported extended property.

Bug fix: Combining characters in Hebrew which were being erroneously split into tokens separated from their bases are no longer being split.

Bug fix: Certain patterns of white-space within Chinese and Japanese tokens no longer cause an internal server error.

Bug fix: The mappings of default Basis POS tags to universal POS tags ("options": {"partOfSpeechTagSet": "upt16"}) have been corrected for Greek.

Bug fix: When fstTokenize is enabled, inputs in Spanish are no longer truncated.

Improved ORGANIZATION type matching: We improved the accuracy of ORGANIZATION name matching by integrating name completion with an internal entity resolution engine.

Example: ソニー株式会社 vs. Sony Corporation

Improved Japanese: We've improved the segmentation of Japanese PERSON names.

Example: スズキタロウ

Spanish token overrides: The Spanish-Spanish token overrides for the PERSON entity type have been expanded.

Example: Francisco vs. Paco

Languages added: The semantic similarity endpoints now also supports Hebrew, Hungarian, Italian, Persian, Portuguese, and Urdu. 

License expiration: The license expiration date has been added to the retrieved API information.

Bug fix: We fixed a bug in the Arabic statistical model and retrained the model.

Latvian support added: Latvian lemmatization is supported. 

Latin-script tokens in Russian: Latin-script regions within Russian inputs are now tokenized and analyzed as English. Example input: “мой новый iPhone”:

Bug fix: When Rosette guessed German compounds they were sometimes lemmatized as verbs but tagged as nouns.

Bug fix: There was a pairwise matching bug where Arabic name A vs Arabic name B produced a different result than Arabic name B vs Arabic name A.

The fragment boundary detector now marks a boundary after any spaces following a fragment boundary delimiter.

Bug fix: There was a sentence break after every Windows newline (i.e. carriage return + line feed).

Bug fix: Multi-script Russian text would have a sentence break each time the script changed.

Bug fix: There were unexpected sentence breaks after some short lines which did not end in whitespace.

Bug fix: Sentence breaks were missing when the sentence break did not align with a token boundary.

Latvian support added: Latvian tokenization is supported. 

Configurable fragment boundaries: The delimiters for the fragment boundary detector are now configurable. A delimiter is restricted to a single character. To configure, edit the value of fragmentBoundaryDelimiters in the rbl-factory-config.yaml file. The string should contain all values that should be recognized as a fragment boundary, including any of the default values you want to keep.

Bug fix: An underscore (U+005F) is no longer treated as a token separator in German when fstTokenize is enabled.

Bug fix: Tokens from multi-script Russian text sometimes had incorrect offsets if fstTokenize was enabled.

Improved address matching: We've boosted the score of tokens that differed only by an inserted or missing space.

Improved address matching: We've added support for additional Spanish and English address abbreviations, such as:

Swedish: We have added entity extraction and linking support for Swedish (swe).

Fragment boundary detection: The fragment boundary detector, which is used to better process data in lists and tables, is now on. Previously it was off. 

New imperative Arabic verbs: We've added the imperative forms of 2000 Arabic verbs to the Arabic lexicon.

Bug fix: In Russian, embedded spaces have been removed from the lemmas of numbers containing spaces. For example, the token "1 234" is now lemmatized to "1234" instead of "1 234".

Bug fix: In Japanese, if a middle dot appeared in the input text immediately before a newline character, it was missing from the tokenized output. Now the middle dot gets a token in the output.

Improved Japanese-English matching for organizations: We've expanded the Japanese-English token overrides for organizations. Overrides explicitly map nicknames, cognates, and variants to improve matching accuracy.

Rosette Enterprise can now support multiple profiles for users that have different processing needs. Each profile consists of a specific set of parameter and configuration settings (e.g., case-insensitive on/off, entity linking on/off) and different data domains (e.g., enabling specific regex and gazetteer files). Custom profiles do not apply to the /address-similarity, /name-similarity, /name-deduplication, and /name-translation endpoints.

Refer to the section Custom Profiles in the Enterprise User Guide for more details.

Dynamic gazetteers: We’ve added a new endpoint, /entities/configuration/gazetteer/add, to add gazetteer entries to the /entities endpoint without having to stop and restart Rosette Enterprise.

Refer to the section Adding Dynamic Gazetteers in the Enterprise User Guide for more details.

Fragment boundary detection: The fragment boundary detector, which is used to better process data in lists and tables, is now on by default. Previously it was off by default.

To turn off fragment boundary detection:

New model for Katakana-Latin name matching: We've added a new deep learning model that improves Katakana-Latin name matching. It is enabled by setting enableSeq2SeqTokenScorer to true. Note that when true, all Japanese names, not just those in Katakana, will be scored with this new model, which may result in lower accuracy for Japanese names in Hiragana or Kanji. Improvements are planned.

New parameter: We've added a new static parameter, katakanaTransliterationOnly, which defaults to false. Setting it to true will cause Japanese names written in Katakana to only be transliterated, not translated. This goes into the jpn_eng: section in the parameter_defs.yaml file.

Refer to the section Name Similarity Configuration Files in the Enterprise User Guide for more details.

Fixed a bug where translating names from Arabic to English could cause a crash.

New endpoint Address Similarity: We’ve added a new endpoint, /address-similarity, which performs a field by field comparison of two addresses and returns a similarity score between 0 and 1. Note that this endpoint is still in LABS and subject to change. Send us your feedback!

Bug fix: Incorporates Chinese tokenization fix from /morphology.

Bug fix: The Chinese tokenizer could create tokens at the end of the input string for temporal nouns without checking that the context was valid for temporal nouns.

Additional token overrides: We’ve added token overrides to the English-English override file and a new fullname override to the Japanese-English fullname override file.

Bug fix: The Rosette install directory name can now contain spaces.