March 2023
Morphological Analysis /morphology/{morphoFeature}
Name Similarity /name-similarity
Turkish support added: Added support for Turkish-English and Turkish-Turkish name matching. We have also added person and organization overrides, stopwords, and language detection to improve matching in Turkish. (RLPNC-6499)
Improved person name matching: RNI-RNT now has the ability to detect given names and surnames in Latin script when the name is of English origin. When the enableAdditionalOnomastics parameter is true, gender mismatch penalty is only applied to the detected given name, as opposed to the first name token in a query. (RLPNC-6719, RLPNC-6720)
Improved Arabic person name matching: The new TRAILING_PATRONYMIC_DELETION match phenomenon provides improved scores for matches which contain a deletion that is caused by truncation of a patronymic. The score of this deletion is controlled by the trailingPatronymicDeletionScore parameter. This only applies to Latin script names of Arabic origin when enableAdditionalOnomastics is true. (RLPNC-6756)
Name Translation /name-translation
Tokenization /tokens
December 2022
Address Similarity /address-similarity
Entity Extraction and Linking /entities
Wikidata refreshed: We've updated the knowledge base data for the provided linking knowledge base. The QID assigned to some extracted entities may differ from previous versions. (RWIki-119, ELK-274, ELK-276)
New currency regex: We've introduced a new option, regexCurrencySplit, that, when set to true, will attempt to split entities extracted with the regex engine of type IDENTIFIER:MONEY into two new entities: IDENTIFIER:CURRENCY_AMT and IDENTIFIER:CURRENCY_TYPE. These two new types represent the amount of the currency (50,000) and the currency type ($), respectively. By default, regexCurrencySplit is set to false. (TEJ-1792)
Tagalog support: We've added case-insensitive NER support for Tagalog. Previously we released a case-sensitive model and we've now added the case-insensitive model as well. (TEJ-1858)
Parameter removed: We've removed the deprecated genre extraction option. This option was used to turn the linker on which has been, and will still be, available by the linkEntities option. The genre option is no longer available in the rex-factory-config.yaml file, as an option in the call, or in the Rosette API bindings (TEJ-1855).
Morphological Analysis /morphology/{morphoFeature}
Ukrainian support added: Tokenization, sentence boundary detection, segmentation user dictionaries, and many-to-one normalization dictionaries are supported for Ukrainian. (ETROG-3594)
Improved part of speech tags: Language-neutral tokens (numbers, symbols, and punctuation) now get part of speech tags in Indonesian, Standard Malay, and Tagalog. (ETROG-3574)
GPU support: Features that use TensorFlow now use a GPU if available. (ETROG-3564)
Emoji support: Emoji 15.0 is now supported. (ETROG-3577)
New option for Katakana: We've added the option joinKatakanaNextToMiddleDot to control whether sequences of Japanese Katakana tokens adjacent to a middle dot should be merged into a single Katakana token. By default, it is true, which matches the behavior in previous versions of /morphology. This option must be enabled in the rbi-factory-config.yaml file. (ETROG-3592)
Bug fix: The Japanese POS tag VN (verbal noun) is now mapped to the UPT-16 POS tag NOUN. It was previously mapped to VERB. (ETROG-3583)
Name Similarity /name-similarity
Improved Japanese organization matching: カンパニー (company) added to the Japanese organization stopwords list. (RLPNC-6545)
Improved Chinese organization matching: We've expanded the list of Chinese stop words for organizations. (RLPNC-6615)
Improved name matching results: When no entityType is specified, the type PERSON will be applied. Previously, the type NONE was applied. (RLPNC-6576)
New parameter for token overrides: We've added a new parameter, overrideSelector, to control which overrides will be considered during querying and matching. Override filenames can now specify a “selector” value which will be matched against this parameter. (RLPNC-6561)
Bug fix: Horizontal tabs are now removed as part of normalization in English. (RLPNC-6541)
Bug fix: Control characters are now removed from Arabic names before matching. (RLPNC-6543)
Bug fix Fixed a case where unexpected name inputs could lead to a null pointer exception. (RLPNC-6634)
Bug fix Fixed an issue where name-similarity could return match scores greater than 1. (RLPNC-6595)
Sentence Tagging /sentences
Tokenization /tokens
Ukrainian support added: Tokenization is now supported for Ukrainian. (ETROG-3594)
New option for Katakana: We've added the option joinKatakanaNextToMiddleDot to control whether sequences of Japanese Katakana tokens adjacent to a middle dot should be merged into a single Katakana token. By default, it is true, which matches the behavior in previous versions of RBL-JE. (ETROG-3592)
September 2022
Entity Extraction and Linking /entities
Tagalog (tgl) support: We've added Tagalog to our list of languages. The following processors are supported: gazetteer, regex, statistical NER, linking. (TEJ-1812, TEJ-1822, TEJ-1785, TEJ-1786)
-
New linking option: We've added a new option for entity linking. When linkMentionMode is set to entities the linker will attempt to link the entities extracted by other processors (regex, gazetters, and the statistical processor) instead of using its own processor to extract entity candidates. Depending on your data, this may provide higher accuracy and speed. (TEJ-1806)
This option has been added to the rex-factory-config.yaml file and is also available as an endpoint option on a per-call basis.
Parameter deprecated: The parameter genre is deprecated and will be removed in the next release.
Bug fix: An exception is no longer emitted when token normalization produces an empty string. (TEJ-1803)
Bug fix: When looking for candidate mentions in text, if there is an overlap between these mentions, the linker now resolves the longest spanning mention before disambiguation. (ELK-277)
Morphological Analysis /morphology/{morphoFeature}
-
Tagalog (tgl) support:
Part of speech (POS) tagging in Tagalog is now supported. (ETROG-3559)
Lemmatization for Tagalog is now supported. (ETROG-3570)
The Tagalog sentence-breaker now recognizes certain abbreviations that end with periods and doesn’t break sentences after them. The tokenizer keeps the period in the token with the rest of the abbreviation. (ETROG-3573)
Indonesian (ind) support: RBL now supports lemmatization for Indonesian, which is the standardized form of Malay spoken in Indonesia. (ETROG-3563)
Standard Malay (zsm) support: RBL now supports lemmatization for Standard Malay, the standardized form of Malay spoken in Malaysia. (ETROG-3563)
-
Bug fix: We fixed a bug in Russian where certain uncommon consonant–vowel sequences in words in the lexicon were incorrectly replaced with more common sequences with different vowel letters. (ETROG-3541)
Example: брошюра
Previously: брошура
Now: брошюра
Name Similarity /name-similarity
Complete CJK Ext A support: We now have full support of CJK Unified Ideographs Extension A. (RLPNC-6324)
Improved Spanish name matching: We have improved Spanish surname detection. (RLPNC-6294)
-
Improved Japanese location matching: All prefectures of Japan are now included in the override list. (RLPNC-6326)
Example: 北海道 vs. Hokkaido Prefecture
Previously: 0.7246
Now: 0.99
Bug fix: The string "luiz arlos da silva bueno" is no longer in the Greek, English, and Vietnamese stop word lists. (RLPNC-6358)
Name Translation /name-translation
Sentence Tagging /sentences
Tokenization /tokens
June 2022
Note
Java 11 and Java 17 are now supported.
Entity Extraction and Linking /entities
Use custom linking knowledge base as an entity extraction gazetteer: We've added the parameter disableLinkerDisambiguation, which was previously only available in the REX SDK. This parameter lets you use the knowledge base as a gazetteer. Each exact match is assigned an ID from the knowledge base without running disambiguation. This disables checking that the context of the document is similar to the context of the knowledge base. (ELK-270)
-
Configure knowledge base linking priority: With multiple knowledge bases it is possible to set the order in which to try linking against each knowledge base. Set the priority in the redactor configuration file (ne_types.xml) (TEJ-1726, TEJ-1754)
Example: The following XML element will set the custom-kb priority higher than the default knowledge base (kb-linker) when linking a PRODUCT entity type:
<ne_type>
<name>PRODUCT</name>
<weight name="kb-linker" value="100" />
<weight name="kb-linker:custom-kb" value="1" />
</ne_type>
relatedEntities renamed to contextWords: When creating a custom knowledge base, the feature contextWords, which was previously called relatedEntities, is required. Context words are language-specific words that are strongly related to the entity. The term relatedEntities has been deprecated. (TEJ-1756)
Bug fix: An error is no longer generated when there are null prefixes in Arabic morphological analyses. (TEJ-1765)
Bug fix: We fixed a bug to enable using noisy_context_vector feature for disambiguation. (ELK-265, ELK-268, ELS-272, TEJ-1776)
Morphological Analysis /morphology/{morphoFeature}
Indonesian support added: Rosette now supports part of speech (POS) tagging in Indonesian. (ETROG-3543)
Malay (standard) support added: Rosette now supports part of speech (POS) tagging in Malay (standard). (ETROG-3545)
Russian lexicon improved: We've added many words related to computer technology to the Russian lexicon. (ETROG-3523, ETROG-3538)
-
Bug fix: In Japanese, negative forms of Ichidan verbs written all in Hiragana are no longer lemmatized to end with “なう”. (ETROG-3534)
Example: Input: くれない
Name Similarity /name-similarity
-
Improved matching of Spanish names and names of Spanish origin: RNI now has a deeper understanding of Spanish surnames. For example: "JOSE JORGE RIOS TORRES" now gets a higher score when matched against "JOSE RIOS" than it does when matched against "JOSE TORRES", since "RIOS" is recognized as the primary surname. (RLPNC-6037)
The following parameters impact how Spanish names are matched:
The new Boolean parameter enableAdditionalOnomastics controls whether to assign a TokenType to allow for multiple Spanish surnames. When set to true, each token is assigned a TokenType, where the TokenType is one of: UNKNOWN, SURNAME, or SURNAME2. It is currently set to true for the spa_eng_PERSON, spa_spa_PERSON and eng:spa_eng_PERSON profiles.
The preexisting parameter surnameTokenTypeWeight now applies only to the TokenType.SURNAME tokens. Its default value was changed from 1 to 1.2.
The new parameter secondarySurnameTokenTypeWeight applies to TokenType.SURNAME2 tokens. Its default value is 0.6.
-
The new parameter crossSurnameMatchPenalty parameter is applied (by simple multiplication) when a TokenType.SURNAME token is scored against a TokenType.SURNAME2 token. Its default value is 0.75.
Example: Pablo Emilio Escobar Gaviria vs. Pablo Escobar
Previously: 0.7945
Now: 0.8309
Example: Pablo Emilio Escobar Gaviria vs. Emilio Gaviria
Previously: 0.7999
Now: 0.7365
-
Improved matching of English organization names: We've added ordinal numbers to the override list for English organizations. (RLPNC-6225)
Example: 1st National Bank vs. First National Bank
Previously: 0.6470
Now: 0.9257
-
Improved Vietnamese name matching: We've expanded the Vietnamese stop word lists for PERSON and ORGANIZATION entity types. (RLPNC-5694)
Example: Chủ tịch Hồ Chí Minh vs. Hồ Chí Minh (translation: President Ho Chi Minh)
Previously: 0.83
Now: 0.99
New parameter for non-phonetic matches: We've added the parameter editDistanceScoreBias to adjust the bias for edit distance scores. Increasing the impact of edit distance scores can improve the match scores of typographical errors and other non-phonetic matches. (RLPNC-6199)
New parameter for organization names: We've added the parameter tokenizeOrganizationsWithNumbers that prevents tokenization of names with numbers within the name. When set to true (default), the number is left within the token and the name will get a higher value from the edit distance scorer. This is desirable if your data contains organization names which intersperse alphabetic and numeric characters or if your data often contains typographical errors with numerals inserted into otherwise valid tokens. When set to false, the number remains within the organization name token. (RLPNC-6200)
Support for Cantonese name transliterations: We've added Jyutping transliterations (Cantonese pronunciation of Chinese names) to the list of readings. The new parameter enableYueReadings enables Jyutping readings. It is set to false by default. To enable Jyupting readings, set enableYueReadings to true. (RLPNC-6239)
-
Improved Japanese-English location name matching: We've expanded the Japanese-English overrides list for location names. (RLPNC-6268)
Example: 大阪府 vs. Osaka Prefecture
Previously: 0.5853
Now: 0.99
Bug Fix: Name similarity will no longer return match scores above 1.0. (RLPNC-6254)
Name Translation /name-translation
March 2022
Notice
Java 8 and Java 9 support is deprecated as of this release.
Address Similarity /address-similarity
Improved Chinese - English address matching: We expanded overrides for ethnic minority regions, particularly from Xinjiang, Tibet, and Inner Mongolia. (RLPNC-6077)
Bug fix: Address matching no longer emits a NullPointerException in certain cases of address matching in which an address contains multiple languages. (RLPNC-6083)
Morphological Analysis /morphology/{morphoFeature}
Spaceless Korean tokenizer: We've added an option to select the spaceless Korean tokenizer in the call. The default tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. To use this tokenizer, set the option modelType to DNN. Previously, this option was not available on a per-call basis. (ETROG-3513)
Name Similarity /name-similarity
Khmer - English added: Khmer - Khmer and Khmer - English are now supported name matching pairs. (RLPNC-5712)
Improved language detection: We've improved language detection for languages that use Han characters (Chinese, Japanese, Korean). (RLPNC-6059)
Improved ORG matching: We've expanded the list of known organization names in our real world ID tables to improve ORG matching in Arabic (ara), Burmese (mya), Chinese (zho), French (fra), German (deu), Greek (ell), Hebrew (heb), Hungarian (hun), Italian (ita), Japanese (jpn), Korean (kor), Portuguese (por), Russian (rus), Spanish (spa), Thai (tha), and Vietnamese (vie). (RLPNC-6090)
New parameter boostWeightAtLeftEnd: We added a new parameter boostWeightAtLeftEnd to increase the weighting of the first token in a name. When setting this parameter, the boostWeightAtRightEnd parameter should not be modified. (RLPNC-6094)
Improved Chinese - English ORG matching: We added override mappings for Chinese numerals in Hanzi to Arabic numbers from zero through twenty-one. (RLPNC-6028)
Bug fix: Pairwise match now works with all languages that have limited language support. Previously, an error was returned for unidentified languages. (RLPNC-6100)
Name Translation /name-translation
Semantic Similarity /semantics/{semanticsFeature}
Tokenization /tokens
Spaceless Korean tokenizer: We've added an option to select the spaceless Korean tokenizer in the call. The default tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. To use this tokenizer, set the option modelType to DNN. Previously, this option was not available on a per-call basis. (ETROG-3513)
February 2022
New
Address Similarity /address-similarity
Version Change - Rosette Server
Version Change - App Server
New Component - App Server
January 2022
New
Version Change - RESTful and App Server
December 2021
New
Version Change - RESTful and App Server
December 2021
New
Version Change - RESTful and App Server
December 2021
ARM Support: Rosette Server can now be installed on Linux ARM and macOS ARM platforms. The deep neural models (DNN) for entities, morphology, sentiment, and relationships are not supported on ARM. The names endpoints (address-similarity, name-similarity, name-deduplication, name-translation) are supported on Linux ARM only.
Usage tracking: When using custom profiles with usage tracking, custom profiles are no longer associated with an application id. All custom profiles are now available to all application ids without requiring file duplication. For backwards compatability with the previous behavior, add the line wrapper.java.additional.250=-Drosapi.feature.CUSTOM_PROFILE_UNDER_APP_ID to the wrapper.conf file.
Address Similarity /address-similarity
Important
If you have any customizations for address stop words or overrides from previous releases, the file names must be renamed to the new file naming convention. The file names now include three letter language codes.
Chinese address matching: We now support Chinese-Chinese and Chinese-English address matching. (RLPNC-5822)
-
Language-specific address override files: Address override files are now language-specific, and the file name must include the language codes. (RLPNC-6032)
Example: English-English state overrides
Example: Chinese-English city overrides
-
Language-specific address stop word files: Stop word files for address matching on text fields (house, road, city, state, country) are now language-specific, and the file name must include the language code (either eng or zho). (RLPNC-6031)
Example: English city stop pattern
Entity Extraction and Linking /entities
Bug fix: Hungarian dates are now extracted correctly. Previously, dates with embedded periods followed by a space were not being extracted. (TEJ-1681)
Bug fix: The entities/info endpoint now returns the complete list of valid entity types. (WS-2360)
Bug fix: The entities/info endpoint no longer lists TEMPORAL types by default for SWEDISH. (TEJ-1687)
Morphological Analysis /morphology/{morphoFeature}
-
Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts in Japanese. (ETROG-3474)
Example: Input: 三・一四
Emojis: U+3030 and U+303D are now tagged as emojis even when not followed by U+FE0F. (ETROG-3478)
Emoji support: We now support the emoji in Unicode 14.0 (ETROG-3476)
Japanese tokenization: In Japanese, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. (ETROG-3475)
Improved POS tags: Many number, punctuation, and symbol characters are now POS-tagged appropriately as numbers, punctuations, and symbols instead of being marked as unknown or some other tag. This applies to all languages with POS tags. (ETROG-3481)
Hungarian improvements: We've added some Hungarian abbreviations and improved sentence boundary detection around Hungarian abbreviations. (ETROG-3479, ETROG-3484)
Bug fix: In Japanese, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)
-
Bug fix: We've reverted two of the POS changes made in version 1.19.0 as they introduced regressions in Chinese and Japanese. (ETROG-3466)
The values are now:
Bug fix: Morphology no longer detects characters as emoji when followed by the text presentation selector (U+FE0E). (ETROG-3480)
Bug fix: In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)
Name Similarity /name-similarity
Basic support for all languages: Name similarity can now match names in any language. Languages which previously would have returned an "unsupported language" error now return a match score. The score is either 1 for a perfect match, or a value based on edit distance. Set the parameter allLanguageSupport to false for backwards compatible behavior to previous versions. (RLPNC-5979)
-
Improved ORG matching: We added real world ID tables or organizational names to improve ORG matching in the following languages: Thai (tha), Greek (ell), Hebrew (heb), Burmese (mya), German (deu), French (fra), Hungarian (hun), Italian (ita), Portuguese (por), Spanish (spa), and Vietnamese (vie). (RLPNC-5986)
Example: "International Astronomical Union" vs. "האיגוד האסטרונומי הבינלאומי"
Previously: score 0.70
Now: score 0.98
-
Neural model for Katakana: We've added a neural-based phonetic matching model to improve Katakana-Latin name matching. To enable the model, set enableSeq2SeqTokenScorer to true in internal_param_defs.yaml file. (RLPNC-5945)
NOTE: This feature is not currently available in Rosette Server.
Improved Hebrew name matching: We've added a rule-based vocalization checker for the statistical-model vocalizer to improve Hebrew-Hebrew and Hebrew-English name matching. (RLPNC-5990)
Name Translation /name-translation
Bug fix: Burmese-English transliteration has been improved by revising the Folk and MLCTS transliteration schemes. (RLPNC-5950)
Bug fix: Hebrew Folk transliteration has been improved, especially for the letters vav and yod. (RLPNC-5916)
Tokenization /tokens
-
Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts in Japanese. (ETROG-3474)
Example: Input: 三・一四
Japanese tokenization: In Japanese, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. (ETROG-3475)
Bug fix: In Japanese, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)
Bug fix: In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)
Version Change - App Server
Version Removed - App Server
Version Removal - RESTful
November 2021
New /custom-profiles endpoint: We've added a new endpoint, rest/v1/custom-profiles. A GET returns a list of all custom profiles known to the server. You can specify an application id header value and it will return all custom profiles specific to that application id.
Installation improvements: The install script and docker configuration update script now update the embedded worker threads along with the original worker thread property.
October 2021
Address Similarity /address-similarity
-
Improved address matching: We've modified the field weight values to provide more accurate address match scores. Weightings were determined by evaluating US and UK address data. (RLPNC-5893)
Example: "85 Court Road Newton Ferrers, Plymouth PL8 1DE1B Devon, England UK” vs “85 Court Road Newton Ferrers PL8 1DE UK"
Previously: Score: 0.73
Now: Score: 0.81
Improved address overrides: Address overrides are now applied to groups of related address fields, instead of just individual fields. Overrides apply when matching any two fields from the same group. (RLPNC-5899)
Entity Extraction and Linking /entities
Name Similarity /name-similarity
ARM64 support: We now support ARM64 processors. (RLPNC-5912)
Burmese transliteration: We added a Basis Technology-created Folk transliteration scheme for Burmese name matching that is similar to how Burmese names are commonly transliterated to English. (RLPNC-5892)
-
Improved Hebrew transliteration: The Hebrew character ח used to be transliterated as “h” in some cases and “kh” in others (if it was followed by a geresh). It is now transliterated as “ch"when not followed by a geresh. The Hebrew character כ used to be transliterated as “h” in some cases and “k” in others (if it has a dagesh). Now, it is transliterated to “ch” in the cases when it used to be transliterated to “h”. (RLPNC-5928)
Example: נחמן
Previously: Nahman
Now: Nachman
Example: מיכל
Previously: Mihal
Now: Michal
August 2021
Address Similarity /address-similarity
-
Improved address matching: We've expanded the override tables for UK, U.S., and Canadian addresses. (RLPNC-5886)
Example: houseNumber<47>road<Albert Street>city<Aberdeen>stateDistrict<Aberdeenshire>postCode<AB25 1XT> vs. houseNumber<47>road<Albert Street>city<Aberdeen>stateDistrict<ABD>postCode<AB25 1XT>
Previously: Score: 0.86
Now: Score: 0.96
-
Bug fix: Overrides for alphanumeric address fields (houseNumber, unit, poBox, postCode) are now being applied. (RLPNC-5863)
Example: “3710 W Martin Luther King Blvd STE #121” vs. “3710 W Martin Luther King Blvd Suite #121”
Previously: Score: 0.833
Now: Score:0.95
Entity Extraction and Linking /entities
Wikidata refreshed: The internal database for Wikidata linking has been refreshed and re-indexed. QIDs for some entities may change from previous versions. (TEJ-1657, TEJ-1658)
New /info endpoint: The entities/info endpoint returns a list of supported entity types available by processor type and language. If you provide a profileId, it returns the list of entity types in the model used by a custom profile. Set the optional parameter perSubsourceSupportedEntityTypes to true to list the entity types by subsource.
-
New option for structured regions: The option structuredRegionProcessingType has been added to specify the type of processing for structured regions. By default, the statistical/DNN model is turned off for structured regions, which increases precision, but may result in reduced recall in structured regions.
To turn on the statistical/DNN model for structured regions, set the option structuredRegionProcessingType to nerModel.
To use the name classifier (LABS) to identify structured regions as PERSON, LOCATION, or NONE entity types, set the option structuredRegionProcessingType to nameClassifier.
The existing option enableStructuredRegion does not determine how structured regions get processed. When enableStructuredRegion is set to true and the input is contentUri (i.e. set to retrieve content from a URL), then HTML lists and tables will be extracted by Tika as structured regions. The value of structuredRegionProcessingType will then determine how those structured regions are processed.
Language Identification /language
Name Similarity /name-similarity
-
Improved Hebrew-English name matching:
We've improved the statistical model. (RLPNC-5842)
-
We changed the default transliteration scheme to FOLK from ISO259-2-1994, which improves matching scores as FOLK more closely matches how people transliterate Hebrew names. (RLPNC-5844)
Example: בִּנְיָמִין גַּנְץ vs. Benjamin Gantz
-
We expanded the token overrides for person entity types. (RLPNC-5845)
Example: אלכס vs. Alexander
-
We added word embeddings for Hebrew organizations. (RLPNC-5837)
Example: ארגון המזון והחקלאות vs. Food and Agriculture Organization
-
Improved Hebrew-Hebrew name matching: We expanded the token overrides for person entity types. (RLPNC-5891)
Example: סולומונתאס vs. סולונאס
-
Improved English-English name matching: We added the token override pair Alex/Aleksandar. (RLPNC-5871)
Example: Alex vs. Aleksandar
-
Improved matching for identifiers: We improved matching and added support for three new subtypes: IDENTIFIER_DRIVERS_LICENSE, IDENTIFIER_LICENSE_PLATE, IDENTIFIER_NATIONAL_ID_NUM, along with IDENTIFIER_GENERIC. (RLPNC-5852)
Example: NH123456789DL vs. NH123456789DN (as IDENTIFIER_DRIVERS_LICENSE entity type)
-
Improved Japanese Segmentation: We've expanded the segmentation dictionary to improve Japanese name segmentation. (RLPNC-5835)
Example: ミロシェヴィッチスロボダン
-
Bug fix: Hebrew tokens containing diacritics are now identified in the override table. (RLPNC-5882)
Example: אֲבִי vs. Abigail
Sentence Tagging /sentences
Tokenization /tokens
June 2021
When using DNN models in the /entities endpoint, a java.lang.OutOfMemoryError would sometimes occur. A configuration setting has been changed, eliminating the error. This only occurred in the 1.19.1 release.
June 2021
Address Similarity /address-similarity
-
Improved handling of postal codes: Better match scores result from this enhancement. (RLPNC-5639)
Example: houseNumber<123>road<Clifton St>city<Cambridge>state<MA>postCode<02140 1234> vs. houseNumber<123>road<Clifton St>city<Cambridge>state<MA>postCode<02140-1234>
Previously: Score: 0.89
Now: Score: 1.0
-
Expanded UK and CA address override tables: We expanded override tables for UK and Canadian addresses. (RLPNC-5607)
Example: houseNumber<100>road<Main Ave>city<Shellbrook>state<Saskatchewan>postCode<S0J 2E0> vs houseNumber<100>road<Main Ave>city<Shellbrook>state<Sask>postCode<S0J 2E0>
Previously: Score: 0.88
Now: Score: 0.96
Entity Extraction and Linking /entities
Morphological Analysis /morphology/{morphoFeature}
New option for tokenizers: We've added a new option, tokenizerType to specify which tokenizer to use. The options alternativeTokenization and fstTokenize are deprecated in favor of tokenizerType. See Tokenizer Types fore more detail. (ETROG-3419)
New Korean tokenizer: We've added a new tokenizer for spaceless Korean input. The previous tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. Activate it by setting tokenizerType to spaceless_statistical. (ETROG-3392)
-
Bug fix: Consecutive punctuation characters are no longer returned as a single token in Chinese when alternativeTokenization is true or tokenizerType is set to spaceless_lexical. Now each character is its own token. (ETROG-3402)
Example: Input: 天津？？
-
Previously:
Token{text=天津}
HanMorphoAnalysis{extendedProperties={}, partOfSpeech=NP, lemma=天津, tagSet=BT_CHINESE}
Token{text=？？}
HanMorphoAnalysis{extendedProperties={}, partOfSpeech=U, lemma=？？, tagSet=BT_CHINESE}
-
Now:
Token{text=天津}
HanMorphoAnalysis{extendedProperties={}, partOfSpeech=NP, lemma=天津, tagSet=BT_CHINESE}
Token{text=？}
HanMorphoAnalysis{extendedProperties={}, partOfSpeech=EOS, lemma=？, tagSet=BT_CHINESE}
Token{text=？}
HanMorphoAnalysis{extendedProperties={}, partOfSpeech=EOS, lemma=？, tagSet=BT_CHINESE}
Name Deduplication /name-deduplication
Name Similarity /name-similarity
Burmese-English added: Burmese-Burmese and Burmese-English are now supported name matching pairs. (RLPNC-5660)
Hebrew-English added: Hebrew-Hebrew and Hebrew-English are now supported name matching pairs. (RLPNC-5339)
Vietnamese-English added: Vietnamese-Vietnamese and Vietnamese-English are now supported name matching pairs. (RLPNC-5687)
-
Improved Chinese-English: Name similarity scores for Chinese-English now leverage English translations from a list of Chinese name translations. (RLPNC-5643)
Example: 汤姆 vs. Tom
Previously: Score: 0.37
Now: Score: 0.99
-
Improved English-English organizations: We expanded the overrides list with numbers and their written form from 1 to 21. (RLPNC-5644)
Example: Channel One Russia vs. Channel 1 Russia
Previously: Score: 0.54
Now: Score: 0.96
Improved name matching for organizations: Name similarity scores for organizations has been improved by adding new frequency models in English, Chinese, Arabic, Japanese, and Russian. (RLPNC-5416)
Updated frequency model: We updated the English frequency model for PERSON entity type by adding birth names from 1920-2019 to the existing model. (RLPNC-5592)
Name Translation /name-translation
April 2021
Entity Extraction and Linking /entities
Language-specific joiner rules: Custom joiner rules can now be language-specific or apply to all languages. (TEJ-178)
New default processing for structured text regions (lists, tables): Because structured text is often just words or phrases, and thus missing the syntactic context that REX was trained on, some REX users would pre-process input text to remove structured regions, on which REX performed poorly. Users no longer have to pre-process the input as now the statistical/DNN model is turned off by default for structured regions. This mode increases precision but may result in reduced recall in these regions. Note, the other REX processors (pattern match, exact match, entity linking) which do not rely on context will continue to analyze the structured regions. To turn on the statistical/DNN model for structured regions, set the parameter structuredRegionProcessingType to nerModel. (TEJ-1502) (TEJ-1502)
New name classifier model for structured regions (LABS): We've added a new model for processing structured regions. Each sentence or fragment (the structured region) is classified as a single entity. The name classifier classifies the entity as PERSON, LOCATION, or NONE. It is disabled by default. It can be specified in the rex-factory-config.yaml file or by using the enableStructuredRegion option in the call. (TEJ-1613, TEJ-1621)
-
New option for structured regions: The option enableStructuredRegion has been added to configure how structured regions are processed. The default value is false. When set to true:
If the input is html, Tika is configured to extract tables and lists, in addition to content.
The name classifier model is used to process structured regions.
Japanese organization gazetteers: The gazetteers for Japanese organizations has been updated to improve extraction of Japanese organizations. (TEJ-1612)
New RBL version: Entity extraction now consumes the latest version of Rosette Base Linguistics (RBL) 7.39.0. (TEJ-1618)
Rosette Training Server (RTS) results: When using REX with Rosette Adaptation Studio (RAS), the results returned by RTS are now preferred by default. (TEJ-1605)
Bug fix: Entities are no longer extracted when they cross a sentence boundary. To enable entity linking across sentence boundaries, set disableApplySentenceBoundaries to true. (ELK-259)
Bug fix: Entities are now checked to ensure they are normalized. (TEJ-1615)
Language Identification /language
Morphological Analysis /morphology/{morphoFeature}
-
Improved Hebrew tokenizer and new analyzer: The Hebrew tokenizer is now more consistent with the tokenizers of other languages. (ETROG-3290)
We've improved tokenization of certain sequences involving digits, periods, and number-related symbols like ⟨%⟩.
We've added additional acronyms and abbreviations to the Hebrew tokenizer. (ETROG-3249)
Double apostrophes are now treated like gershayim. (ETROG-3249)
The nfkcNormalize option is now supported for Hebrew.
Normalized characters: Normalized half-width and full-width characters are processed the same as their counterparts. (ETROG-3351)
-
Bug fix: Hebrew tokens containing a geresh are now tokenized properly. Previously, only the part up to the geresh would be returned as the token, and the part after the geresh would sometimes be considered a suffix. Now all the characters of the token are returned as the token. ETROG-3290)
Example: מע'רב
-
Previously:
Token{text=מע'}
MorphoAnalysis{extendedProperties={hebrewPrefixes=[], hebrewSuffixes=[]},
partOfSpeech=noun, lemma=מע', tagSet=MILA_HEBREW}
MorphoAnalysis{extendedProperties={hebrewPrefixes=[מ, ב], hebrewSuffixes=[ר, ב]},
partOfSpeech=numeral, lemma=70, tagSet=MILA_HEBREW}
-
Now:
Token{text=מע'רב}
MorphoAnalysis{extendedProperties={com.basistech.rosette.bl.hebrewPrefixes=[],
com.basistech.rosette.bl.hebrewSuffixes=[]}, partOfSpeech=unknown, lemma=מע'רב,
tagSet=MILA_HEBREW}
February 2021
Entity Extraction and Linking /entities
Bug fix: A previous entity offset alignment issue involving very short regex-captured entities followed by \r is fixed.
Bug fix: We reverted some changes to Korean tokenization from the 1.18.0 release. You may see minor differences in extracted entities in Korean when comparing results from the 1.18.0 and 1.18.1 releases.
Morphological Analysis /morphology/{morphoFeature}
Greek lexicon: The Greek lexicon has additional nouns.
New Greek disambiguator added: The new Greek disambiguator is more accurate, but slower. The new disambiguator is enabled by default. To use the old disambiguator, set alternativeGreekDisambiguation to true.
-
Greek disambiguation improved: Certain Greek forms are now disambiguated to prefer a modern analysis over an archaic analysis. alternativeGreekDisambiguation must be set to false, which is the default.
Example: δείξε
Deprecated classes: The classes BufferWordBreaker and WordBreakResults have been deprecated.
Emoji normalization: Morphology no longer normalizes certain emoji zero-width joiner (ZWJ) sequences to U+1F48F KISS, U+1F491 COUPLE WITH HEART, and U+1F46A FAMILY, to be consistent with Unicode’s efforts to make emoji more gender-neutral by default.
-
Bug fix: The Greek guesser now handles tokens with non-alphanumeric characters.
Example: Start+
Previously: POS tags: possible PROP, ADJ, NOUN
Now: POS tag: FM (foreign word)
Bug fix: We reverted some changes to Korean disambiguation from the 1.18.0 release as the changes introduced new disambiguation errors. You may see minor differences in Korean disambiguation when comparing results from the 1.18.0 and 1.18.1 releases.
Name Similarity /name-similarity
Character normalization improved: We've improved character normalization for all supported languages. All languages except for Japanese, Korean, and Arabic-script languages now have NFKC normalizations performed. Japanese, Korean, and Arabic-script languages have NFKD normalizations performed.
-
Japanese ORGANIZATION type matching improved: We've improved the accuracy of ORGANIZATION matching for Japanese-English and Japanese-Japanese names by expanding the stop word list for the ORGANIZATION entity type.
Example: コダック合同会社 vs. Kodak Limited
-
Better entity resolution for Russian ORGANIZATION type matching: We've improved Russian-English and Russian-Russian name matching by adding support for Russian organizations in the entity resolution engine.
Example: Ура́льские авиали́нии vs. Ural Airlines
-
Expanded stop word list for Russian ORGANIZATION type matching: We've improved the accuracy of ORGANIZATION matching for Russian-English and Russian-Russian names by expanding the stop word list for the ORGANIZATION entity type.
Example: Балтийский федеральный университет имени Иммануила Канта vs. Immanuel Kant Baltic Federal University
-
Korean ORGANIZATION type matching improved: We've improved Korean-English and Korean-Korean name matching by adding support for Korean organizations in the entity resolution engine.
Example: 현대자동차 vs. Hyundai Motor Company
-
Bug fix: We fixed a bug where Arabic-Arabic name matching was returning a low score in some cases for names that were seemingly very similar.
Example: عبدلخالق الحوثي vs. عبدالخالق الحوثي
Version Change - Restful and Embedded
Added - Restful and Embedded
December 2020
Entity Extraction and Linking /entities
Wikidata refreshed: The internal database for Wikidata linking has been refreshed and re-indexed. QIDs for some entities may change from previous versions.
Bug fix: The sample for the SQLite-kb-connector now works correctly. Runtime issues with SQLite dependencies have been corrected.
Bug fix: Extraction no longer fails when a custom processor returns a NULL annotator; a warning is issued instead.
Bug fix: Mentions normalized by a custom processor are no longer ignored.
Bug fix: Newlines \r and \r\n are now handled correctly.
Morphological Analysis /morphology/{morphoFeature}
Deprecated option: The alternative tokenization option deliverExtendedAttributes is now deprecated. Previously it delivered an unsupported extended property.
Bug fix: Combining characters in Hebrew which were being erroneously split into tokens separated from their bases are no longer being split.
Bug fix: Certain patterns of white-space within Chinese and Japanese tokens no longer cause an internal server error.
-
Bug fix: The mappings of default Basis POS tags to universal POS tags ("options": {"partOfSpeechTagSet": "upt16"}) have been corrected for Greek.
Previously: COSUBJ mapped to CONJ, ORD mapped to ADJ, and POSS mapped to DET
Now: COSUBJ maps to ADP, ORD maps to NUM, and POSS maps to PRON
Bug fix: When fstTokenize is enabled, inputs in Spanish are no longer truncated.
Name Similarity /name-similarity
-
Improved ORGANIZATION type matching: We improved the accuracy of ORGANIZATION name matching by integrating name completion with an internal entity resolution engine.
Example: ソニー株式会社 vs. Sony Corporation
-
Improved Japanese: We've improved the segmentation of Japanese PERSON names.
Example: スズキタロウ
-
Spanish token overrides: The Spanish-Spanish token overrides for the PERSON entity type have been expanded.
Example: Francisco vs. Paco
Semantic Similarity /semantics/{semanticsFeature}
Languages added: The semantic similarity endpoints now also supports Hebrew, Hungarian, Italian, Persian, Portuguese, and Urdu.
API information /info
Version Change - RESTful and Embedded
Removed - RESTful and Embedded
New Version Added - RESTful and Embedded
New Version Added - RESTful
October 2020
Language Identification /language
Bug fix: Rosette now correctly identifies the primary language of short documents which contain small fragments of a language in another script. Previously, the language of the fragments might be erroneously detected as the primary language. The lengths of the document's script regions are now taken into account when identifying the primary language.
Morphological Analysis /morphology/{morphoFeature}
-
Bug fix: Sentence breaks are now correct when there are two line breaks and fragmentBoundaryDetection is enabled.
Example: "a very very very very long line\nshort\n\n"
-
Bug fix: In Hebrew, lemmas starting or ending with spaces now have the spaces removed.
Example: "אאורקה "
Previously: "אאורקה "
Now: "אאורקה"
Rosette Server On-Premise Only
Morphological Analysis /morphology/{morphoFeature}
Decompose noun compounds: Support for decomposeCompounds has been expanded to Danish, Dutch, German, Hungarian, Korean, Norwegian (Bokmål, Nynorsk), and Swedish. The default value is true.
September 2020
Address Similarity /address-similarity
-
New per-call option - parameters: Requests can now specify a parameters object to update parameter values for an individual request. Any non-static parameter can be changed. To specify, add "parameters": {"parameterName":"value"} to the call.
Example:
{
"address1": {
"houseNumber": "122",
"street": "main st",
"postCode": "02222"
},
"address2" : {
"houseNumber": "123",
"street": "main st",
"postCode": "02222"
},
"parameters": {
"postCodeAddressFieldWeight": "2.0",
"stateAddressFieldWeight": "0.5"
}
}
-
Improved address matching: Address matching has been improved by improving the normalization of the postal code address field.
Example: 71-75 Shelton street London WC2H9JQ vs. 71 SHELTON STREET LONDON WC2H 9JQ
Previously: 0.6239
Now: 0.8542
Entity Extraction and Linking /entities
Improved phone number recognition: Regular expressions for phone number extraction have been improved and now extract more phone number patterns.
Bug fix: We've partially fixed a problem in Japanese ORG extraction where sometimes the model extracts multiple ORG entities or includes non-related adjacent tokens.
Name Similarity /name-similarity
-
New per-call option - parameters: Requests can now specify a parameters object to update parameter values for an individual request. Any non-static parameter can be changed. To specify, add "parameters": {"parameterName":"value"} to the call.
{
"name1": {
"text": "Kraft Services",
"entityType": "Organization"
},
"name2": {
"text": "Kraft Srvs",
"entityType": "Organization"
},
"parameters": {
"deletionScore": "0.2"
}
}
-
Expanded English stop words: Organization name matching has been improved by the addition of English stop words.
Example: SUNY Canton vs. State University of New York at Canton
Previously: 0.8340
Now: 0.8713
-
Improved Spanish, Arabic, and Korean organization matching: By adding the use of word embeddings, we've improved organization name matching when one or both of the names are in the specified language.
Spanish-English example: Astilleros y Talleres del Noroeste vs. Shipyards and Workshops of the Northwest
Previously: 0.4558
Now: 0.8838
Korean-English example: 아시아나 항공 vs. Asiana Airlines
Previously: 0.7208
Now: 0.8672
Arabic-English example: الاتحاد العالمي للحفاظ على الطبيعة والمصادر الطبيعية vs. International Union for Conservation of Nature and Natural Resources
Previously: 0.3263
Now: 0.7106
Rosette Server On-Premise Only
Entity Extraction and Linking /entities
Joiner runs before redactor: The joiner now runs before the redactor by default, providing more flexibility and control over the joiner results. Set runJoinerPostRedactor to true to run the joiner after the redactor.
Bug fix: We fixed a bug where sometimes a null pointer exception was returned when the custom processor and the linker had overlapping results.
Bug fix: Custom processors can now only modify the entity and metadata sections of the ADM. Previously, any modification could be made which could override annotation data.
New Version Added - Proxy Server
Version Change - Embedded and Restful
August 2020
Morphological Analysis /morphology/{morphoFeature}
Greek coverage expanded: POS tags and lemmas are now recognized for some Greek words previously not identified. (ETROG-3225)
Bug fix: Fragment detection now counts tokens correctly to determine short lines. This mostly impacts languages without spaces: Chinese, Japanese, and Thai. (ETROG-3177)
-
Bug fix: Tokens with digits are now analyzed by for the Greek guesser. (ETROG-3231)
Previously: "HDMI1" defaulted to possible PROP, ADJ, NOUN POS tags
Now: "HDMI1" gets FM POS tag
-
Bug fix: Russian perfective verbs are now lemmatized correctly. Previously some were lemmatized to their imperfective counterparts' lemmas or other incorrect lemmas. (ETROG-3112)
Example: "разложу" where "разложу" is perfective and its lemma is "разложить". Its imperfective counterpart’s lemma is "раскладывать"
Previously: Two analyses: one lemmatized to "раскладывать", the other to "разлагать"
Now: One analysis, lemmatized to "разложить"
-
Bug fix: German lemmas that consist of a separable prefix and a noun are now correctly capitalized. (ETROG-3235)
Example: Input "Mitbehandlung"; "mit" is a separable prefix
-
Bug fix: In Hebrew, terminal combining characters are no longer getting split into their own tokens. (ETROG-3224)
Example: "1" (keycap)
Previously: Tokenized to two tokens, <U+0031 DIGIT ONE> <U+20E3 COMBINING ENCLOSING KEYCAP>.
Now: Tokenized to one token, "1"
Name Similarity /name-similarity
-
Arabic improvements: New PERSON and ORGANIZATION stopwords have been added, improving Arabic-English and Arabic-Arabic name matching.
Example for PERSON entity type: محمد vs. نبي محمد
Example for ORGANIZATION entity type: بنك الأهلي التجاري vs. ال البنك الأهلي التجاري
Name Translation /name-translation
Hebrew - English added: Hebrew to English is now a supported translation language pair. Rosette supports the Hebrew transliteration standards ISO259_2_1994 and ICU (which refers to the default Hebrew transliterator implemented by ICU and is based on the UNGEGN standard for geographic names), but defaults to FOLK. This Basis Technology-created transliteration scheme is more useful than the other more academic standards (ISO259_2_1994 and ICU) as it closely resembles how people in the real world write Hebrew names with Latin characters.
Semantic Similarity /semantics/{semanticsFeature}
Rosette Server On-Premise Only
Entity Extraction and Linking /entities
Morphological Analysis /morphology/{morphoFeature}
New short line parameter: The option maxTokensForShortLine has been added to configure how many tokens can be in a line for it to be considered short for fragment boundary detection. The default value is 6. (ETROG-3179)
Greek time abbreviations: The time abbreviations "π.μ." and "μ.μ." are now identified and annotated in Greek. The option fstTokenize must be set to true. (ETROG-3226)
-
Bug fix: Whitespace-delimited fragment boundaries are no longer skipped when they fall within tokens. This only occurred when fstTokenize was enabled and in some languages. (ETROG-3159)
Example: "1\n234" (embedded newline within the number string)
This example assumes fstTokenize is enabled and the language is French.
-
Bug fix: In Hebrew, tokens with an unknown part of speech are no longer assigned the part of speech of one of their prefixes. This only applies when the option guessHebrewPrefixes is set to true. (ETROG-3221)
Example: "ומפיפרנו"
Version Change - Embedded and Restful
Version Change - Application Server
Added - Application Server
July 2020
Rosette Server On-Premise Only
July 2020
Rosette Server On-Premise Only
Dynamic custom profiles Changes to a custom profile can now be loaded without restarting Rosette Enterprise. To dynamically change a custom profile, delete the profile directory from the disk and create a new one with the changes. The new directory can have the same name as the deleted profile directory.
June 2020
Address Similarity /address-similarity
-
Support for unfielded addresses: Addresses no longer have to be provided as separate field components.
{
"address1": "The Book Club 100-106 Leonard St Shoreditch London EC2A 4RH, United Kingdom",
"address2": "The Book Club 100-108 Leonard St Shoreditch London EC2A 4RH, UK"
}"
-
Added field overrides: More overrides (matches) have been added for address fields, improving the match scores. Overrides explicitly map nicknames, cognates, and variants to improve matching accuracy.
Example: England and UK have been added as overrides
{
"address1": {
"house": "Ffrwdgrech Industrial Estate",
"road": "Ffrwdgrech Rd",
"city": "Brecon",
"country": "UK",
"postcode": "LD3 8LA"
},
"address2": {
"house": "Ffrwdgrech Industrial Estate",
"road": "Ffrwdgrech Rd",
"city": "Brecon",
"country": "England",
"postcode": "LD3 8LA"
}
}
Previously: 0.86
Now: 0.95
-
Improved support for misfielded address components: We've improved scores by checking for matches in different fields to catch where address components were input into the wrong field.
Example: Washington and D.C. are matched even though they were put into different fields
{
"address1": {
"houseNumber": "1600",
"road": "Pennsylvania Ave N.W.",
"city": "Washington",
"state": "D.C.",
"postcode": "20500"
},
"address2": {
"houseNumber": "1600",
"road": "Pennsylvania Ave N.W.",
"city": "D.C.",
"state": "Washington",
"postcode": "20500"
}
}
Entity Extraction and Linking /entities
Hebrew improvements: Entity extraction has improved Hebrew normalization. The disambiguator can now identify prefixes removed from the entity's normalized form. Improvements are a result of Hebrew morphology enhancements.
Bug fix: A new line character in a regex (\n) will now also match carriage returns (\r) and a combination of both (\r\n).
Morphological Analysis /morphology/{morphoFeature}
-
Hebrew improvements: Hebrew tokens that have prefixes but not stems now get tagged the correct part of speech. Previously, they got the POS tag "unknown".
Example: “ה” from the string “ה70”
-
Bug fix: Minimally-qualified emoji are no longer split apart.
Example: The emoji for "man tipping hand" (<U+1F481, U+200D, U+2642>: 
)
Previously: U+1F481 and <U+200D, U+2642> (2 tokens)
Now: <U+1F481, U+200D, U+2642> (1 token)
-
Bug fix: Capitalized common nouns are no longer detected as verbs.
Example: The noun "Service" from the phrase "Price and Quality of Service"
Name Similarity /name-similarity
-
Improved Arabic-English and Arabic-Arabic matching: Many enhancements have improved the name similarity scores when matching Arabic names with Arabic or English names.
-
Token alignment Improved token alignment between names in Arabic and English
Example: Aung San Suu Kyi vs. أون سان سو تشي
Previously: 0.4544
Now: 0.7261
-
New statistical model A new Arabic-English statistical model has been added.
Example: Debby Ryan vs. ديبي رايان
Previously: 0.6114
Now: 0.9202
-
Gender identification We've added gender identification for Arabic names
Example: Mario Savinova vs. ماريا سافينوفا
-
Language model The weighting of tokens in Arabic names has been improved.
Example: Margaret Thatcher vs. مارجريت تاتشر
Previously: 0.6299
Now: 0.8182
-
Initials We've added support for initials and initialisms in Arabic.
Example: J vs. جمال
Previously: 0.4106
Now: 0.8256
-
Japanese-English Name similarity scores for organizations has been improved by expanding the list of organization name equivalents mapped between Japanese-English (aka, token overrides).
Example: 中国建設銀行 vs. China Construction Bank
Previously: 0.8701
Now: 1.0
Rosette Server On-Premise Only
Address Similarity /address-similarity
Windows support: To parse unfielded addresses, address similarity depends on the jpostal binding for the open source libpostal library. Though jpostal is not officially supported on Windows, our tests have shown it to function as expected. Please contact support if you discover any issues.
Entity Extraction and Linking /entities
More complete sample files: The sample files to build the SQLite connector described in the Custom Knowledge Base Connectors section now includes all files required to build with Maven. The configuration to run the connector with Rosette Enterprise is now provided as well.
Bug fix: Confidence scores for entity linking now use the same scale, whether linking to Wikidata or a custom knowledge base. Previously, the confidence scores given for links to custom knowledge bases were much lower than those calculated for the Wikidata knowledge base.
Morphological Analysis /morphology/{morphoFeature}
-
Hebrew improvements: When guessHebrewPrefixes is true, unrecognized Hebrew tokens will now get analyses with and without potential prefixes. Previously, they would only get analyses with potential prefixes.
Example: Token: "ומפיפרנו"
-
Previously: 2 analyses:
hebrewPrefixes=[ו] lemma=מפיפרנו
hebrewPrefixes=[ו, מ] lemma=פיפרנו
-
Now: 3 analysis:
hebrewPrefixes=[ו] lemma=מפיפרנו
hebrewPrefixes=[ו, מ] lemma=פיפרנו
hebrewPrefixes=[] lemma=ומפיפרנו
Added - Embedded and Restful
April 2020
/supported-languages (all endpoints)
Entity Extraction and Linking /entities
Hebrew accuracy improvement: We re-trained the statistical model, adding an annotated finance dataset. The accuracy is improved for the finance genre and remains the same for the news genre.
New DNN model for Hebrew: We added a new deep neural network model for Hebrew. It performs better than the statistical model on both news and finance genres.
Hebrew normalization: For more accurate downstream processing, we have removed prefixes from normalized Hebrew output, except for the definite article.
Morphological Analysis /morphology/{morphoFeature}
Hebrew proper nouns: We added more proper nouns to the Hebrew lexicon.
German professions: We added more German professions to the German lexicon.
Additional emoji support: Emoji hair components are now lemmatized.
Spanish performance improvements: Spanish disambiguation is now faster.
Tokenization /tokens
Unicode 13.0 emojis: Unicode 13.0 emoji sequences are now tokenized.
Bug fix: We fixed a bug where low surrogates were stripped from the ends of tokens in Hebrew.
Known issue: Minimally-qualified emoji zero-width joiner (ZWJ) sequences are no longer recognized as emoji. We will fix this in the next release.
Rosette Server On-Premise Only
Entity Extraction and Linking /entities
Geo-coordinates regex: We added supplemental regex support for ISO 6709 geo-coordinates for all languages.
Linking knowledge bases: You can now load multiple custom knowledge bases and prioritize which knowledge base to try linking with first. Edit the kbs parameter in the rex-factory-config.yaml file.
Redactor improvement: Redactor weights can now be configured for specific subsources used by processors. For example, the entity linker can link to multiple knowledge bases, each of which is considered a subsource. These subsources can have their own redactor weights to adjudicate linking conflicts among the subsource.
Custom profile with custom knowledge base: You can now define a custom knowledge base in a custom profile.
New license file: Linking to custom knowledge bases is now licensed separately. If you link to one or more custom knowledge bases, you will need to install a new license file. Linking to wikidata does not require a new license file.
Name Similarity /name-similarity
Tokenization /tokens
Added - Embedded and Restful
Added - Proxy Server and Application Server (Custom Endpoint Example)
February 2020
Rosette Platform Changes
Entity Extraction and Linking /entities
Morphological Analysis /morphology/{morphoFeature}
Latvian support added: Latvian lemmatization is supported.
-
Latin-script tokens in Russian: Latin-script regions within Russian inputs are now tokenized and analyzed as English. Example input: “мой новый iPhone”:
Bug fix: When Rosette guessed German compounds they were sometimes lemmatized as verbs but tagged as nouns.
Name Similarity /name-similarity
Sentence Tagging /sentences
The fragment boundary detector now marks a boundary after any spaces following a fragment boundary delimiter.
Bug fix: There was a sentence break after every Windows newline (i.e. carriage return + line feed).
Bug fix: Multi-script Russian text would have a sentence break each time the script changed.
Bug fix: There were unexpected sentence breaks after some short lines which did not end in whitespace.
Bug fix: Sentence breaks were missing when the sentence break did not align with a token boundary.
Tokenization /tokens
Rosette Enterprise On-Premise Users Only
Sentence Tagging /sentences
Configurable fragment boundaries: The delimiters for the fragment boundary detector are now configurable. A delimiter is restricted to a single character. To configure, edit the value of fragmentBoundaryDelimiters in the rbl-factory-config.yaml file. The string should contain all values that should be recognized as a fragment boundary, including any of the default values you want to keep.
Bug fix: An underscore (U+005F) is no longer treated as a token separator in German when fstTokenize is enabled.
Bug fix: Tokens from multi-script Russian text sometimes had incorrect offsets if fstTokenize was enabled.
LABS Usage Tracking: Rosette Enterprise can now track Rosette calls by app-id, profileID, endpoint, and language. See the Rosette User Enterprise User Guide for more information. Note that this feature is still in LABS and subject to change.
LABS Custom Endpoints: Rosette Enterprise now supports creation of custom endpoints that combine business logic, custom workflows, and Rosette endpoints into a single call. See the Rosette User Enterprise User Guide for more information. Note that this feature is still in LABS and subject to change.
Name Similarity /name-similarity
The deep learning model for Katakana-Latin name matching has been disabled in this release. The setting enableSeq2SeqTokenScorer must be set to false, which is the default. It will be available again in a future release.
Added - Proxy Server and Application Server (Custom Endpoint Example)
December 2019
Rosette Platform Changes
Address Similarity /address-similarity
-
Improved address matching: We've boosted the score of tokens that differed only by an inserted or missing space.
-
Improved address matching: We've added support for additional Spanish and English address abbreviations, such as:
calle : cl
camino : cno, cmno
Entity Extraction and Linking /entities
Morphological Analysis /morphology/{morphoFeature}
Fragment boundary detection: The fragment boundary detector, which is used to better process data in lists and tables, is now on. Previously it was off.
New imperative Arabic verbs: We've added the imperative forms of 2000 Arabic verbs to the Arabic lexicon.
Bug fix: In Russian, embedded spaces have been removed from the lemmas of numbers containing spaces. For example, the token "1 234" is now lemmatized to "1234" instead of "1 234".
Bug fix: In Japanese, if a middle dot appeared in the input text immediately before a newline character, it was missing from the tokenized output. Now the middle dot gets a token in the output.
Name Similarity /name-similarity
Improved Japanese-English matching for organizations: We've expanded the Japanese-English token overrides for organizations. Overrides explicitly map nicknames, cognates, and variants to improve matching accuracy.
Rosette Enterprise On-Premise Users Only
System Requirements: JDK 11 is now supported.
Custom Profiles
-
Rosette Enterprise can now support multiple profiles for users that have different processing needs. Each profile consists of a specific set of parameter and configuration settings (e.g., case-insensitive on/off, entity linking on/off) and different data domains (e.g., enabling specific regex and gazetteer files). Custom profiles do not apply to the /address-similarity, /name-similarity, /name-deduplication, and /name-translation endpoints.
Refer to the section Custom Profiles in the Enterprise User Guide for more details.
Entity Extraction and Linking /entities
-
Dynamic gazetteers: We’ve added a new endpoint, /entities/configuration/gazetteer/add, to add gazetteer entries to the /entities endpoint without having to stop and restart Rosette Enterprise.
Refer to the section Adding Dynamic Gazetteers in the Enterprise User Guide for more details.
Morphological Analysis /morphology/{morphoFeature}
-
Fragment boundary detection: The fragment boundary detector, which is used to better process data in lists and tables, is now on by default. Previously it was off by default.
To turn off fragment boundary detection:
Name Similarity /name-similarity
New model for Katakana-Latin name matching: We've added a new deep learning model that improves Katakana-Latin name matching. It is enabled by setting enableSeq2SeqTokenScorer to true. Note that when true, all Japanese names, not just those in Katakana, will be scored with this new model, which may result in lower accuracy for Japanese names in Hiragana or Kanji. Improvements are planned.
-
New parameter: We've added a new static parameter, katakanaTransliterationOnly, which defaults to false. Setting it to true will cause Japanese names written in Katakana to only be transliterated, not translated. This goes into the jpn_eng: section in the parameter_defs.yaml file.
Refer to the section Name Similarity Configuration Files in the Enterprise User Guide for more details.
Removed from Embedded and RESTful
Apache Aries Blueprint API v1.0.1
Apache Aries Blueprint CM v1.0.8
Apache Aries Blueprint Core Compatibility v1.0.0
Apache Aries Blueprint Core v1.6.2
Apache Aries Proxy API v1.0.1
Apache Aries Proxy Impl v1.0.5
Apache Aries Util v1.0.0
Apache Aries Util v1.1.1
Apache Commons CLI 1.2
Apache Commons Lang 3.3.2
Apache Commons Weaver v1.1
Apache HTTPComponents v4.4.1
Apache Geronimo OSGI Factory Registry v1.1
Apache Geronimo Xbean v4.1
Apache Groovy v2.4.7
Apache Jakarta Regexp v1.4
Apache Service Mix Specs Activation API 1.1 v2.4.0
Apache Service Mix Specs JAXB API 2.2 v2.4.0
Apache Service Mix Specs SAAJ API 1.3 v2.4.0
args4j 2.32
ASM v5.0.2
ASM v5.0.3
ASM v5.0.4
Jackson Datatype Joda 2.4.5
Java API for RESTful Web Services v1.1.1
Java Mail API 1.4.4
Jettison v1.3.7
JRuby v9.1.6.0
Metro XML Information Set v1.2.13_1
Reflections v0.9.10_3
Swagger Annotations 1.5.7
Woodstox v3.0.1
Woodstox v3.1.0
Woodstox v3.1.4
Woodstox v4.0.0
Woodstox 4.0.5
Woodstox 5.0.1
Wordnik Swagger Annotations 1.5.3-M1
Removed from Embedded Only
Added to Embedded and RESTful
November 2019
Rosette Enterprise On-Premise Users Only
Name Translation /name-translation
October 2019
Rosette Platform Changes
Address Similarity /address-similarity (LABS)
New endpoint Address Similarity: We’ve added a new endpoint, /address-similarity, which performs a field by field comparison of two addresses and returns a similarity score between 0 and 1. Note that this endpoint is still in LABS and subject to change. Send us your feedback!
Entity Extraction and Linking /entities
Morphological Analysis /morphology/{morphoFeature}
Name Similarity /name-similarity
Rosette Enterprise On-Premise Users Only
Morphological Analysis /morphology/{morphoFeature}
Table 1. Open Source Package Changes Embedded and RESTful
Package |
Old Version |
New Version |
License |
org.apache.commons:commons-collections4 |
4.0 |
4.4 |
Apache 2.0 |
September 2019
Rosette Enterprise On-Premise Users Only
Name Translation /name-translation
August 2019
Note
Breaking Changes
Be sure to check below if you use the following features:
Entity linking with DBpedia
Language identification with Malaysian
Entity extraction and linking with Rosette Enterprise On-Premise
Rosette Platform Changes
Entity Extraction and Linking /entities
New Feature - PermID Linking (LABS): We have augmented QID (Wikidata) linked entities by also linking them to Thomson Reuters Permanent Identifiers (PermIDs) when PermIDs are included in the Wikidata entry. To access these identifiers, add {"options": {"includePermID": true}} to your call. For more information, see the Features and Functions. Note that this feature is still in LABS and subject to change. Please try it out and send us your feedback!
Wikidata update: We’ve refreshed and re-indexed the internal database for Wikidata linking. As a result, QIDs for some entities may have changed from previous versions.
DBpedia update: A QID can now be associated with more than one DBpedia subtype. A new option parameter, includeDBpediaTypes, has been added. A new response field, dbpediaTypes, is a list that can contain one or more string values. The original response field, dbpediaType, will continue to return a single string: the first value on the list. The original option and response field are now deprecated and will be removed in a future release. If you are using DBpedia values, please migrate to the new fields.
Bug fix: We fixed a bug in entity extraction and linking where it would crash when the modelType option is set to DNN, the linkEntities option is set to true, and the Entity Linker and DNN mentions overlap.
Language Identification /language
Malaysian update: Language identification now returns zsm for Standard Malay (Malaysian) when using both long string and short string algorithms. Previously, zsm was returned from the short string algorithm and msa (Malay macrolanguage) from the long string algorithm. This may require updates if you have code looking for the language code msa.
Increased short string language coverage: Language identification's standard and short-string algorithms now have identical language coverage (except for transliterated Arabic-script languages). Added support for Albanian, Bulgarian, Catalan, Croatian, Estonian, Icelandic, Kurdish (Arabic script), Kurdish (Latin script), Latvian, Lithuanian, Macedonian, Polish, Serbian (Cyrillic script), Serbian (Latin script), Slovak, Slovenian, Somali, Tagalog, Ukrainian, Urdu (Arabic script), Uzbek (Cyrillic script), Uzbek (Latin script), and Vietnamese to the short string language identification algorithm.
Multilingual short string support: For sufficiently short strings, language identification will now use its short-string algorithm in multilingual mode.
Morphological Analysis /morphology/{morphoFeature}
Polish update: Some Polish words ending in -cku, -ska, or -sku are now (following a different convention) lemmatized to forms ending in -cki or -ski. They had previously been lemmatized to their surface forms.
Bug fix: The Japanese POS tag NE remained NE and was not converted correctly to UPT-16 (Universal POS tagset). NE is now correctly converted to the UPT-16 PART.
Bug fix: The French POS tag CONJQUE was converted to UPT-16 CONJ. It is now converted to the more appropriate SCONJ.
Bug fix: Chinese punctuation was tagged as GUESS when alternativeTokenization was disabled. Chinese punctuation is now tagged as PUNCT or EOS.
Name Similarity /name-similarity
-
Improved Chinese-Chinese name matching: We now give greater weight to names whose romanization is identical.
-
Improved name matching and translation of Japanese by normalizing small Katakana characters into their full-sized counterparts.
Previously: Small Ka in 田中ヵ wasn’t being transliterated and the following readings were generated instead (tanaka ヵ||denchuu ヵ||tana ヵ||tianzhong ヵ||jeonjung ヵ||tanaka ヵ).
Now: Small Ka in 田中ヵ now gets normalized to 田中カ and the following readings get generated (tanaka ka||denchuu ka||tana ka||tianzhong ka||jeonjung ka||tanaka ka).
-
Improved name matching and translation in Chinese by normalizing Extension A characters.
Previously:㨗 in 㨗報 used to get ignored and the following readings were generated (bao||mitsugi||cubbon).
Now:㨗 in 㨗報 maps properly to its variant and now the following readings get generated (jie bao||shou mitsugi||cubbon).
Semantic Vectors /semantics/vector
Token frequency used for document vectors all languages: For all languages, document vectors are now calculated using token frequency information. This information gives more weight to less common words, which are more significant to the overall meaning of the document. Previously, only English vectors were weighted this way.
Rosette Enterprise On-Premise Users Only
The default configuration of the Entity Extraction and Linking endpoint has fewer options enabled to optimize performance. The endpoint is now aligned with the configuration of the REX-JE SDK. Rosette Cloud has a different configuration, providing a fully-functional demonstration environment. For more details on how to set the options back to the previous settings, see the section “Entity Extraction and Linking Default Configuration” in the Rosette Enterprise User Guide.
New default settings:
Entity linking is disabled (linkEntities: false)
No supplemental regular expressions are loaded (supplementalRegularExpressionPaths null)
Pronominal resolution is not enabled (resolvePronouns: false)
caseSensitivity: caseSensitive
Table 2. Open Source Package Changes Embedded and RESTful
Table 3. Open Source Packages Deleted from Embedded
Package |
Version |
Checker Qual |
2.5.2 |
Error Prone |
2.1.3 |
J2ObjC |
1.1 |
JSR 305: Annotations for Software Defect Detection |
3.0.2 |
Mojohaus Animal Sniffer Annotations |
1.14 |
June 2019
Rosette Platform Changes
Entity Extraction and Linking /entities
Japanese Update: Japanese data regex now includes the new era 令和 (Reiwa), which started May 1, 2019.
Bug fix: We've fixed additional cases where Japanese characters were wrongly normalized into their simplified Chinese equivalents in entity linking. This issue was also addressed in the 1.13.0 release.
Bug fix: We've added missing parameter files for linking models, which may improve accuracy.
Morphological Analysis /morphology/{morphoFeature}
Updated lexicons: We've made various small updates to the lexicons of German, English, and Swedish.
Improved Arabic analyzer: The Arabic analyzer will attempt to replace leading hamzated alefs with plain alefs for unrecognized tokens, to see if the version with the plain alef is recognized.
Improved Hebrew normalization: U+2019 RIGHT SINGLE QUOTATION MARK and U+201D RIGHT DOUBLE QUOTATION MARK are now normalized to U+0027 APOSTROPHE and U+0022 QUOTATION MARK in Hebrew, to support their use as geresh and gershayim.
Bug fix: We've fixed a bug which occurred when Rosette encountered a Hebrew token consisting of multiple prefixes without a base. In this case, Rosette returned an incomplete surface form, i.e., only containing the first prefix. For example, the surface form of the token “מה” should be “מה” but was being returned as “מ”. With the bug fix the surface form returned now is “מה”.
Bug fix: We’ve fixed a problem where closing parentheses, brackets, and braces that follow URLs were merged into the URLs.
Bug fix: The disambiguator is now more likely to select analyses for @mentions, email addresses, hashtags, and URLs over other analyses.
Bug fix: When the Hebrew tokenizer encounters a character not used in Hebrew immediately following a character used in Hebrew, it now starts a new token. Formerly, it would delete that character and any following characters up to the next token separator (e.g. whitespace).
Bug fix: Hebrew tokens consisting of multiple prefixes without a base are now tagged with the part of speech “unknown”, to match single-prefix tokens. Previously, multi-prefix tokens were not tagged with any part of speech.
Rosette Enterprise On-Premise Users Only
Morphological Analysis /morphology/{morphoFeature}
Bug fix: When fstTokenize is enabled, we’ve fixed a problem where Russian hyphenated words that end in numbers, like “Аполлона-11”, were not tagged as DIG. They are now tagged with the same parts of speech they had before Rosette API 1.12.2
May 2019
Rosette Enterprise On-Premise Users Only
To minimize the size of your Rosette Enterprise installation, the entity extraction (rex-root) and semantic similarity (tvec-root) components are now shipped by language. The name of the language specific files contain the three letter ISO-639 language code, indicating which language is supported by the file.
Entity extraction is shipped with one base file and one or more language-specific files.
Example:
rex-root-<version>.tar.gz
rex-root-<version>-eng.tar.gz for English language files
rex-root-<version>-deu.tar.gz for German language files
Semantic Similarity is shipped with one file per language.
Example:
The Rosette Enterprise installer has been updated and will automatically install all components as required, based on your license.
April 2019
Rosette Platform Changes
Categorization /categories
Entity Extraction and Linking /entities
Bug fix: We’ve fixed a problem in Japanese where entity names that include the middle dot were not being handled correctly. Entity names that include a middle dot are no longer split into two entities.
Bug fix: We’ve fixed a problem in Japanese entity linking where, in some cases, Japanese characters were being replaced with Chinese characters.
Bug fix: We’ve fixed a problem where, in some cases, entities were mislabeled when the includeDbPediaTypes option was not flagged.
Morphological Analysis /morphology/{morphoFeature}
New Hebrew disambiguator: We’ve added a new default disambiguator (perceptron) for Hebrew. Use the disambiguatorType option to enable another disambiguator (DNN or dictionary). For example, to return to the previous default, add {"options": {"disambiguatorType": "DNN"}} to your call.
Bug fix: We’ve fixed an error where some Chinese tokens included spurious white space characters.
Bug fix: We’ve fixed a problem where extremely long tokens (thousands of characters) would slow down the tokenizer.
-
Bug fix: We’ve fixed a problem where Polish tokens that can appear in multiword expressions were lemmatized to full expressions, even when the full expression wasn’t in the input.
Bug fix: We've fixed a problem where the non-final components of Russian compound words with more than one hyphen were not lemmatized correctly.
Name Similarity /name-similarity
Sentiment Analysis /sentiment
Expanded English support: We can now process English with a case-insensitive model by providing (uen) as the input language. Be aware that when using DNN for the modelType the accuracy of the results may be lower than when analyzing standard, sentence-cased, English input.
Rosette Enterprise On-Premise Users Only
Sentiment Analysis /sentiment
Document-only analysis: Entity-level sentiment analysis can now be turned off, allowing document-level sentiment analysis only. See the Rosette Enterprise User Guide for more information.
Expanded Language support: Users of the Rosette Text Classification Field Training Kit can now train custom sentiment analysis models in any language supported by the tokenization endpoint (for document-level analysis) or the entity extraction and linking endpoint (for entity-level analysis). For more information on the training and configuration procedure, see the Rosette Field Training Kit documentation.
Open Source Changes
February 25, 2019
Entity Extraction and Linking /entities
Morphological Analysis /morphology/{morphoFeature}
Bug fix: We've fixed a bug where the Persian lemmatizer did not add lemmas to the first analyses of many tokens, especially verbs.
Bug fix: We've fixed a bug where after some sequences of 4096 characters, containing mostly white space and at most one token, any following tokens had incorrect original offsets.
Semantic Similarity /semantics/{semanticsFeature}
Bug fix: We've fixed a bug where capitalized tokens could return an out-of-vocabulary token-level embedding instead of the embedding consistent with their lowercase form.
Bug fix: Previously, the Semantic Vectors endpoint did not always return vectors of a consistent length. Now, returned vectors will always be normalized to have a length of one.
Rosette Enterprise On-Premise Users Only
Morphological Analysis /morphology/{morphoFeature}
Categorization /categories
New factory configuration option: We’ve added a new factory configuration option, maxResults. This option can be used to cap the number of results returned. By default, all results exceeding the score and confidence thresholds (if set) will be returned.
Open Source Changes
Deleted from Rosette Enterprise Embedded
Deleted from Rosette Enterprise Restful
January 31, 2019
Semantic Similarity /semantics/{semanticsFeature} (LABS)
Note that the Semantics Similarity features are still in LABS and subject to change. Send us your feedback!
-
New endpoint Similar Terms: We've added a new endpoint, /semantics/similar, which uses text vectors to generate multilingual related terms with numerical similarity scores for any input word(s) in Arabic, English, Chinese, German, Japanese, North or South Korean, Russian, or Spanish. For more information, see the Features and Functions.
Input Term: spy returns
Spanish
{"term":"espía","similarity":0.61295485},
{"term":"cia","similarity":0.46201307},
{"term":"desertor","similarity":0.42849663},
{"term":"cómplice","similarity":0.36646274},
{"term":"subrepticiamente","similarity":0.36629659}
German
{"term":"Deckname","similarity":0.51391315},
{"term":"GRU","similarity":0.50809389},
{"term":"Spion","similarity":0.50051737},
{"term":"KGB","similarity":0.49981388},
{"term":"Informant","similarity":0.48774603},
Japanese
{"term":"スパイ","similarity":0.5544399},
{"term":"諜報","similarity":0.46903181},
{"term":"MI6","similarity":0.46344957},
{"term":"殺し屋","similarity":0.41098994},
{"term":"正体","similarity":0.40109193},
Semantic Vectors: The /text-embedding endpoint has been renamed to /semantics/vector. While the /text-embedding endpoint will remain accessible through April, we encourage you to migrate as soon as possible to avoid missing any important updates.
Entity Extraction and Linking /entities
Improved linking confidence: We've updated the linking confidence calculation and thresholds to improve accuracy.
Supported languages: We've removed xxx from the list of languages returned when using /entities/supported-languages.
Bug fix: We’ve fixed a bug where the salience score was not always returned for entities with pronominal mentions, when requested.
Bug fix: We've fixed a bug where some later entity mentions that were chained to the first mention of a given entity were not always properly returned.
Bug fix: We've fixed a bug where sometimes a null pointer exception was returned when resolving pronouns.
Morphological Analysis /morphology/{morphoFeature}
Bug fix: We’ve fixed a bug where some English words were automatically getting tagged as proper nouns when capitalized.
Bug fix: We’ve fixed a bug in English where “people” was not being properly lemmatized to “person”. It now has the lemma candidate “person” when appropriate.
Bug fix: We’ve fixed a bug where ordinal numbers and comparative adjectives in English like “second” and “lower” were analyzed as verbs.
Name Similarity /name-similarity
-
Hungarian: We’ve added support for multi-letter initials in Hungarian, improving Hungarian name matching.
-
Japanese: We’ve improved the accuracy of matching between katakana and kanji versions of Japanese organization names.
-
Chinese: We’ve improved the accuracy of matching Chinese organization names.
Previously: The similarity score for 松下能源(上海)有限公司 and Panasonic Energy (Shanghai) Co., Ltd was 0.58.
Now: The similarity score for 松下能源(上海)有限公司 and Panasonic Energy (Shanghai) Co., Ltd is 0.82.
Rosette Enterprise On-Premise Users Only
We've decreased warm-up time by only loading licensed languages when Rosette is set to pre-warm.
We’ve added the ability for on-premise users to have more control over configuration options when using custom-trained models.
We've reduced the minimum memory requirements to 16GB of RAM for the entity extraction and linking, sentiment analysis, and topic extraction endpoints.
We’ve improved the efficiency of the initial load time for the entity extraction and linking endpoint.
Client Bindings
Open Source Changes
December 10, 2018
Entity Extraction and Linking /entities
Korean: We've improved the accuracy of Korean extraction, largely through better handling of Josa (postpositions) and compound words.
Entity Linking: We've added support for entity linking to Wikipedia for both the top level types (PERSON, LOCATION, ORGANIZATION, ETC.) as well as the over 700 DBpedia types (see full list here) in the remaining 16 languages supported by entity extraction. This is in addition to the languages currently supported by entity linking: Chinese, English, Japanese, and Spanish.
Morphological Analysis /morphology/{morphoFeature}
Hebrew disambiguation: We’ve improved analysis in Hebrew by adding disambiguation, a mechanism for more accurately choosing which of several candidate analyses is provided in the response. For Hebrew only, we've added an option disambiguatorType to select which disambiguator is used. The values are DNN for the TensorFlow-based deep neural network model and dictionary for the dictionary-based model. The default is dictionary. To enable the DNN disambiguator, add {"options": {"disambiguator": "DNN"}} to your call.
Persian lemmatization: We’ve added lemmatization support to Persian.
Name Similarity /name-similarity
Rosette Enterprise On-Premise Users Only
-
The minimum system requirements for running Rosette Enterprise have changed for some use cases. This is a result of providing entity linking for 16 additional languages in this release. We now support entity linking for all 20 languages supported by entity extraction.
The entity extraction and linking, sentiment analysis, and topic extraction endpoints require significant memory allocation. If using these endpoints, the new minimum memory requirements are:
For all other endpoints, the minimum memory requirements are:
Rosette Enterprise is now available as a Docker container. The images are available on Docker Hub. A Basis shipment, containing a license file and a docker-compose file customized to your licensed endpoints, is still required.
October 29, 2018
Morphological Analysis /morphology/{morphoFeatue}
-
Chinese: We’ve added the word “百度” to the Chinese lexicon. This only has an effect when modelType is set to default, which is its default value.
-
German: The part of speech of the acronyms “MAN” and “MIT” is now NOUN in German, instead of falling back to the parts of speech of the unrelated words “man” and “mit”.
-
Spanish: We’ve improved Spanish part-of-speech tag and lemma disambiguation.
Name Similarity /name-similarity
-
New Model: We’ve added a new model for increased accuracy when matching Hungarian names to other Hungarian names.
-
Bug Fix: Fixed a bug involving duplicate readings produced when transliterating Chinese names.
September 24, 2018
Entity Extraction and Linking /entities
Improved Chinese accuracy: We've replaced the underlying tokenizer to improve accuracy in Chinese.
Improved Hungarian accuracy: We've updated our pattern match extractors in Hungarian. This improves accuracy for the MONEY and DATE types.
Known issue: We've fixed a bug where entity mentions were being miscounted if the calculateSalience option was set to true.
Morphological Analysis /morphology/{morphoFeature}
Improved Dutch disambiguation: We've improved Dutch part-of-speech disambiguation.
Improved Japanese and Chinese lemma support: In the last release, most Japanese and Chinese tokens did not have lemmas when modelType was set to default. Now, such tokens have lemmas equivalent to their surface forms.
August 27, 2018
Rosette Enterprise On-Premise Users Only
New per-endpoint licensing: Endpoints are now activated directly from your installed license. The endpoints.yaml file has been removed from the installation.
New Enterprise User guide: We’ve added a user guide (rosette-enterprise-user-guide-1.11.0.pdf), that provides new content and replaces the files rosette-api-on-premise-install-guide-1.11.0.txt, overview.md, and Rosette_API_Embedded_User_Guide.pdf.
Simplified installation for macOS and Linux: The installation for both RESTful and embedded Java has been simplified for macOS and Linux. Installation for Windows has not changed, but detailed installation notes are now included as part of the new Enterprise User Guide.
Name Changes: We are continuing to consolidate and simplify our branding. Rosette API On-Premise is now Rosette Enterprise. We’ve made changes to the documentation and license names to reflect.
Rosette Platform Changes
New supported languages sub-endpoints: For all endpoints (excluding Name Similarity, Name Translation, and Name Deduplication), Rosette now provides a GET /rest/v1/<endpoint>/supported-languages method that returns that endpoint's supported languages and scripts. See the Features and Functions or the Interactive Docs for more information.
Updated bindings: We've updated our CSharp and Java bindings. Be sure to get the latest version (1.11.0) to take advantage of all the new features and improvements!
Name Deduplication
Categorization
Multilabel categorization: Rosette can now return multiple category labels per document. For more information, see the Features and Functions. To return only a single category label per document, set the {"options": {"singleLabel": true}}. For more information, see the Features and Functions.
Text Embedding
New languages: The text embedding endpoint now supports Russian, North Korean, South Korean, and Arabic.
Individual token embeddings: We can now return embeddings for individual input tokens. To enable per-token embeddings, add {"options": {"perToken": true}} to your call.
Response modifications: We've made changes to the text embeddings endpoint's response structure. Document-level embeddings now have their own dedicated slot embeddings and will no longer appear in documentMetadata. Please note that this is a breaking change, contact Rosette support for more information.
Entity Extraction and Linking
New feature - DBpedia Types (LABS): We've added over 700 new entity types to the Entity Extraction and Linking endpoint, drawn from the DBpedia ontology. To access these entity types, add {"options": {"includeDBpediaType" = true}} to your call. You'll notice more than 10 additional macro types in the type field as well as the all new DBpediaType field. For more information, see the Features and Functions. Note that this feature is still in LABS and subject to change. Send us your thoughts!
Better accuracy: We've improved the recall of Rosette's entity linking across all supported languages.
New language: Entity extraction now supports Hungarian.
Known issue: MONEY, PHONE NUMBER, and URL types are not extracting properly in Hungarian. This will be fixed in the September 2018 patch release.
Language Identification
Morphological Analysis
New algorithm for Chinese and Japanese: We've added a new algorithm for Chinese and Japanese morphological analysis. Prior to version 1.11.0, the default algorithm was perceptron. To return to the old model, add {"options": {"modelType": "perceptron"}} to the body of your call.
Norwegian lemmatization: We’ve expanded the lemma dictionaries for Norwegian, both Bokmål and Nynorsk.
Improved English and Spanish disambiguation We've improved the accuracy of lemmatization and part of speech tagging in both English and Spanish.
Bug fix: We've improved handling of formatting characters in German.
Bug fix: We’ve fixed a bug where the Hebrew POS tag wPrefix was not converted to UPT-16.
Tokenization
New algorithm for Chinese and Japanese: We've added a new algorithm for Chinese and Japanese tokenization. Prior to version 1.11.0, the default algorithm was perceptron. To return to the old model, add {"options": {"modelType": "perceptron"}} to the body of your call.
Bug fix: We’ve fixed a bug where in Catalan, in which Rosette did not tokenize after an apostrophe in cases where the apostrophe marks a token boundary.
Bug fix: We've fixed a big in Japanese, in which Rosette did not recognize 々 as a Japanese character, so it was considered its own token.
Name Similarity
New language: Name similarity now supports matching between Hungarian and English names.
Improved language-of-origin detection: We've improved the detection of language-of-origin of Japanese names written in Katakana.
June 25, 2018
Name Similarity
Entity Extraction and Linking
Morphological Analysis
Bug fix: We've fixed a bug where some components of compound German words were incorrect when the surface form of the component could be either a noun or a verb.
Bug fix: We've fixed a bug where the Hebrew parts of speech tags did not use UPT-16, the POS tag set used by the other languages.
Bug fix: We've fixed a bug where returned email addresses and URLs could contain control or whitespace characters.
Bug fix: We've fixed a bug where returned Hebrew tokens could contain control characters or nothing but default ignorable characters.
Bug fix: We've fixed a bug where Chinese, Japanese, and Thai tokens could contain control characters.
Tokenization
Name Deduplication
May 30, 2018
Sentiment Analysis
Syntactic Dependencies
Entity Extraction and Linking
Improved Hebrew Entity Extraction: We've improved Hebrew entity extraction by removing superfluous prefixes from extracted entities.
Improved Confidence Scores: We've improved statistical model confidence scores to provide a more effective tradeoff between precision and recall. Please note, this update may cause results to change. If you have set a threshold based on entity confidence scores, please evaluate to ensure optimal performance.
Improved Entity Normalization: Social media characters such as "@" and "#" are removed from a Mentions normalized string. Offsets to the original string data field remain the same.
Morphological Analysis
Bug fix: Previously, the Dutch disambiguator would always choose analyses whose lemmas matched their surface forms, even for very rare lemmas; now the more common lemma will be returned. For example, schepen can be a singular noun with the lemma schepen, but it is more likely to be a plural noun with the lemma schip.
April 23, 2018
Entity Extraction and Linking
New deep neural network processor (in BETA): We've added an alternative entity extraction processor, which can be used in place of the standard statistical extractor. The new processor employs a deep neural network that improves accuracy up to 7% and error rate up to 32%. It is available for English, Arabic, and Korean. To enable this processor, provide DNN for the modelType. Example: {"content": "your_text_here", "options": {"modelType": "DNN"}}
Morphological Analysis
New language support: We've added support for lemmatizing Catalan, Estonian, Serbian, and Slovak text.
Bug fix: Previously, tokens could be empty or contain only invisible characters. Such tokens will no longer be returned.
Language Identification
Sentiment Analysis
Name Translation
Name Similarity
New language support: Rosette now supports matching of Greek names written in Greek script to English names written in Latin script and other Greek names written in Greek script.
Accuracy improvements: We have improved match scores and segmentation rules for Arabic, Western Farsi, and Japanese names.
March 27, 2018
Morphological Analysis
Bug fix: We’ve improved our handling of tokens consisting of numbers and Latin characters, such as serial numbers, in Korean. Previously these tokens were decompounded into multiple morphemes.
Bug fix: We’ve added the lower case Russian word “интернет” (“internet”) to the dictionary, which was previously only present in title case.
Bug fix: We’ve improved our handling of tokens containing an apostrophe immediately followed by a digit in languages like French and Italian, like “all'M5S”. Previously, the apostrophe would be parsed as its own token.
Bug fix: We’ve improved our German analysis by taking better advantage of context clues, and now return more accurate results, especially for uncommon words.
Bug fix: We’ve improved our handling of English and German words in all-caps. Previously, these words were assumed to be proper nouns, even though all-caps may simply denote emphasis.
Transliteration
Entity Extraction and Linking
February 17, 2018
Morphological Analysis
German disambiguation: We’ve improved our German disambiguator for lemmas and part-of-speech tags to be more sensitive to capitalization, particularly for single word inputs.
Bug fix: Previously, German definite articles (der, die, das, den, dem, and des) meaning the were lemmatized inconsistently. They are now all lemmatized to the masculine singular nominative form, der.
Bug fix: In some languages, an apostrophe may mark a token boundary, like in the Italian phrase all'M5S. Previously the token boundary was incorrectly omitted when the following token contained a digit. This issue has been rectified and M5S will be properly tokenized.
Name Similarity
February 6, 2018
Entity Extraction and Linking
Currency support: We’ve added several additional currency symbols to the regex, including the Turkish Lira (₺), the Pound Sterling (₤), and the Euro (€).
Bug fix: We’ve fixed a bug that caused hexadecimal number strings to be incorrectly extracted as products.
Name Translation
Name Similarity
January 16, 2018
Topics
Salience scores: We've added salience scores for keyphrases and concepts to indicate how relevant an extracted concept or keyphrase is to the overall content of a text. You now have the option to filter out results below a desired threshold value: {"content": "your_text_here", "options": {"keyphraseSalienceThreshold": value, "conceptSalienceThreshold": other_value}}.
Short string support: We've improved our concept extraction logic, and now support concept extraction for short input strings, i.e. texts less than 280 characters long.
Name Deduplication
Sentiment Analysis
New feature: We've added the option to use an experimental alternative deep neural network (DNN) sentiment model for English: {"content": "your_text_here", "options": {"modelType": "DNN"}}. The new model will produce different results, which may be more accurate than the current support vector machine (SVM) model, depending on your data. As it is experimental, we are particularly interested in getting user feedback. On-premise users of Rosette API should review the new system requirements in install-guide.txt before using this option.
Entity Extraction and Linking
Entity offsets returned: Entity mention offsets are now returned by default. Offsets can be used to locate the exact surface forms of an extracted entity in the document text.
Korean improvements: We’ve significantly improved the accuracy of entity extraction results across all entity types in Korean.
Confidence scores: Confidence scores for entities extracted using Rosette’s statistical processor, as well as all linked entities, will now be returned by default. Confidence scores allow Rosette to return the most accurate results, particularly for entity linking. To change this behavior, set {"content": "your_text_here", "options": {"calculateConfidence": false}}.
Language Identification
Score changes: We’ve rescaled the confidence scores returned by the language identification endpoint based on customer feedback. The ranking of language candidates will not change, but the scores themselves will be higher. If you currently filter language identification results based on a confidence threshold, you will need to reset that threshold to maintain parity with previous versions.
Name Translation
Name Similarity
Thai support: Rosette now supports matching of Thai names to English names and other Thai names.
Accuracy improvements: We have improved match scores for Arabic names (persons, locations and organizations) as well as for Chinese and Japanese organizations.
November 13, 2017
Entity Extraction and Linking
Bug Fix: This release addresses a bug whereby entity linking confidence scores were not being returned when requested. Confidence scores for entities resolved to Wikipedia entries will now be returned when using the following option: {"content": "your_text_here", "options": {"calculateConfidence": true}}
October 23. 2017
Topic Extraction
New Endpoint: Topic extraction We've added a topic extraction endpoint that identifies the key ideas of an input text. For a given input, the endpoint will return two lists: Keyphrases, a list of phrases extracted directly from the text, and Concepts, a list of phrases which do not have to be explicitly mentioned in the input.
LABS
Sentiment Analysis
Entities
Salience Scoring: Rosette can now return salience scores, which indicate whether an entity is important to the overall scope of the document. Turn on the scores by adding an option to the request: {"content": "your_text_here", "options": {"calculateSalience": true}}
Linking Confidence Scoring: Rosette can also now return Linking Confidence scores, which represent the degree of certainty of the link between an in-document entity mention and its linked QID. It may be used for thresholding and removal of false positives. Linking Confidence scores for entities identified by our linker and assigned with a QID are now available by adding an option to the request: {"content": "your_text_here", "options": {"calculateConfidence": true}}
July 26, 2017
Name Deduplication
New Endpoint: Name Deduplication We've added a name deduplication endpoint that identifies similar names within a list. The endpoint accepts a list of names, organizes the list into clusters of unique names, and assigns each cluster with an id number. It then returns those ids to the user.
June 22, 2017
Entity Extraction
Bug fix: This release addresses a backward compatibility issue between the latest Rosette API and older versions of our Java binding that affected Rosette's ability to return entity confidence scores. Confidence scores for entities identified by our statistical extractor are now available by adding an option to the request: {"content": "your_text_here", "options": {"calculateConfidence": true}}
June 14, 2017
Transliteration for Arabizi
Arabic Sentiment Analysis
Relationship Extraction
Entities
Tokenization
New support for emoticons, emoji, @mentions, hashtags, URLs, and email addresses: These special characters and character combinations are now kept together as a single token in all languages, greatly improving the accuracy of analysis further downstream.
Morphological Analysis
Improved accuracy for English and Spanish: For this release, we updated our English and Spanish dictionaries. We also introduced new, more advanced disambiguation models for these languages, which help Rosette to correctly determine a given word’s part of speech. For example, words like “object” can be either a noun (“this is an object”) or a verb (“I object!”).
Lemmatization and normalization of emoticons, emoji, @mentions, hashtags, URLs, and email addresses: Rosette now normalizes and lemmatizes these special characters and character combinations to streamline analysis.
Improved decompounding for Dutch: Dutch language text is now decompounded more accurately, Dutch text is now decompounded more accurately, producing better tokens for search enhancement and other applications.
March 23, 2017
Relationship Extraction
Improved Accuracy of Corporate Relationships: Improvements made to the identification of relationships between corporations. The relationships involved are: ORG-SUBSIDIARY-OF, ORG-COLLABORATORS, ORG-ACQUIRED-BY and ORG-PROVIDER-TO.
Removed the ORG-PARTNERSHIPS Relationship: The ORG-PARTNERSHIPS relationship is now subsumed under ORG-COLLABORATORS and is no longer extracted as an independent relationship.
Entity Extraction and Linking
Improved Linking Accuracy via Inclusion of New Context Features: The statistical model for entity linking includes features that measure the vector space similarity between an entity context and the Wikipedia contexts of its potential linking targets. The new features result in higher F-Scores across all supported languages.
Entity Linking in Japanese, Chinese and Spanish: Entity linking to Wikidata with QIDs for Japanese, Chinese and Spanish text is supported.
Removed Long Text Linking: Entity linking to Wikidata (with QIDs) for long texts is removed, which, as a result, removed entity linking capabilities in Arabic.
Text Embedding
Language Identification
Name Matching
Japanese Improvements: Rosette API now has better support for Japanese name matching. This includes the new use of word embeddings, which are used to match words with similar semantic meaning, as well as improved Japanese name segmentation.
January 10, 2017
Targeted Relationship Extraction
New Endpoint Functionality: The /relationships endpoint now returns targeted relationships, as opposed to the former open relationships, as its default extracted relationships. Targeted relationships are specifically between two entities, and are labeled by a certain relationship type. You can see the former open relationships by setting the option of "discoveryMode" to "true".
/entities/linked REMOVED
Removed Deprecated Endpoint: The /entities/linked endpoint, previously deprecated, is now completely removed. All functionality is available through the /entities endpoint. You will receive a 404 when calling /entities/linked.
Entity Extraction
Social Media Linking in Japanese and Chinese: Our fast short text entity linker to Wikidata is now available for Japanese and Chinese.
Removal of long text entity linking: Our long text entity linker has been replaced by our fast short string entity linker. You will now see entity linking results from our short string linker by default. This removes linking support for Arabic.
Additional Language Support: The entity extractor now supports Vietnamese.
CJK Support for Names
Name Translation and Similarity CJK Improvements: The /name-similarity and /name-translation endpoints now support matching and translating between Japanese-Chinese, Japanese-Korean, and Korean-Chinese. Japanese accuracy was improved significantly.
Text Embeddings Improvement
Japanese Sentiment Analysis
October 27, 2016
Syntactic Dependencies (NEW)
Relationship Extraction
Modality Returned: We've also added a "modality" field to Rosette's Relationship Extraction. Modality is the semantic context of the possibility or necessity of the relationship; the values can be “assertion”, “negation”, “uncertainty”, “opinion”, or “question”.
Starter Plan (NEW)
New $99 API Plan: For a limited time, we’re offering a special Starter plan. $99/month gets you 40,000 Rosette API calls. Want to dive deep into Rosette but don’t need a whole 100,000 calls? This plan is for you.
September 15, 2016
Text Embedding (NEW)
Sentiment Analysis
Additional entities: We changed the /sentiment endpoint to return the sentiment of all entities discovered by Rosette, including Person, Location, Organization, Date, Time, and more entity types.
Entity Extraction
Global changes
Concurrency header: We added the X-RosetteAPI-Concurrency header to return the number of concurrent calls allowed on your plan. If you are receiving 429 errors, Too Many Requests, then Contact us for greater concurrency.
July 21, 2016
Global changes
Input genre: The genre field is available for /entities and /entities/linked to indicate the input is from social media. Specifying genre=social-media does not affect the output of the other endpoints. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences.
Entity Linking
Entity Extraction
Entity endpoints unified: We combined the /entities and /entities/linked endpoints into one endpoint, /entities. Rosette now returns the entity mentions and the entityId, if available. The entityId replaced the indocChainId. The output of /sentiment has not changed.
Entities Linked deprecated: We deprecated the /entities/linked endpoint. It is still available, but we recommend that you adapt your applications to the new /entities endpoint.
Additional entities: Rosette now extracts more entity types: Date, Time, Longitude and Latitude, and Distance.
Japanese entityId: We added support for linking entities in Japanese (jpn) text to their entityId.
Spanish social media: We added support for extracting entities from social-media in Spanish language documents, using the genre field.
Malay entities: We added support for extracting entities in Malay (msa).
Error code
Sentiment Analysis
NEW Binding Release - Ruby and R bindings
June 20, 2016
Bindings
Ruby: We added the Ruby binding to the gray column to the right and on Github. There is a Ruby gem available as well.
R: We added the R binding to the gray column to the right and on Github.
cURL examples: We changed the shell examples in the gray column on Features and Functions to be cURL code examples.
May 10, 2016
Entity Linking
Social input: We added a request field, "genre": "social-media", to speed up and improve the accuracy of linking Person, Location, Organization and Product entities in social media posts. English input only.
March 29, 2016
Global changes
Language used: We added a Response Header, X-RosetteAPI-ProcessedLanguage, to return the language used by Rosette for processing the call. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences
requestId moved: We moved the requestId object from the JSON response body to the Response Header as X-RosetteAPI-Request-Id. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences, /name-translation, /name-similarity
Rosette API Key: We changed the user_key header’s name to X-RosetteAPI-Key. The user_key header is deprecated. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences, /name-translation, /name-similarity
unit parameter removed: We removed the optional unit request parameter. All input will be handled as a doc. Applies to: /entities, /entities/linked, /relationships, /sentiment, /morphology
Base64: We removed support for Base64 encoding. You can submit binary files as a multipart/form-data call type. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences
Entity Extraction
Relationship Extraction
Accuracy mode: We removed the optional accuracy mode. All input will be processed with the precision accuracy mode, so Rosette will return a precise list of accurate relationships.
Explanations removed: The explanations value has been removed from the response object.
Categorization
Sentiment Analysis
Entity sentiment: We added support for entity-level sentiment analysis. The JSON response for the /sentiment endpoint now includes two objects – document and entities. See the interactive documentation for examples of this new response.
Neutral result: We added a neutral label for documents and entities with a neutral sentiment.
Short strings: Rosette will automatically process short and long content with our proprietary algorithm for sentiment analysis.
Explanations removed: The explanations value has been removed from the response object.
Morphological Analysis
Added language support: We added language support for Dari, Persian, Urdu, and Western Farsi for Parts-of-Speech Tags.
Universal POS Tags: We return Universal Parts-of-Speech Tags for all supported languages.
Tokens list: Rosette returns parallel lists of tokens, lemmas, compound components, parts-of-speech tags, and Han-readings. If a token does not have a lemma, compound component, POS tag, or Han-reading, or if the language is not supported, then Rosette will return “null” in that list.
Name Translation
Renamed to /name-translation: To clarify the endpoint’s function, we renamed /translated-name to /name-translation. /translated-name is no longer available.
Removed result layer: Within the response to /name-translation and /name-similarity endpoints, we removed the result layer so the results are in the response object. Also applies to: /name-similarity
TargetScheme requires uppercase: For advanced users who would like to specify a targetScheme, the scheme must be submitted in uppercase.
Name Matching