Rosette Cloud Release Notes

• The log files 500-exception.log and rosapi.log are not created automatically. Contact Support if you need these log files.

• Bug fix: Rosette now correctly identifies the primary language of short documents which contain small fragments of a language in another script. Previously, the language of the fragments might be erroneously detected as the primary language. The lengths of the document's script regions are now taken into account when identifying the primary language.

• Bug fix: Sentence breaks are now correct when there are two line breaks and fragmentBoundaryDetection is enabled.

  Example: "a very very very very long line\nshort\n\n"

  • Previously: 2 sentences

    "a very very very very long line\n"
    "short\n\n"

  • Now: 1 sentence

    "a very very very very long line\nshort\n\n"

• Bug fix: In Hebrew, lemmas starting or ending with spaces now have the spaces removed.

  Example: "אאורקה "

  • Previously: "אאורקה "

  • Now: "אאורקה"

• Bug fix: We've fixed a problem where the endpoints might not see the correct license.

• New per-call option - parameters: Requests can now specify a parameters object to update parameter values for an individual request.  Any non-static parameter can be changed. To specify, add "parameters": {"parameterName":"value"} to the call.

  Example:

  {
    "address1": {
      "houseNumber": "122",
      "street": "main st", 
      "postCode": "02222"
    },
    "address2" : {
      "houseNumber": "123",
      "street": "main st", 
      "postCode": "02222"
    },
    "parameters": {
      "postCodeAddressFieldWeight": "2.0",
      "stateAddressFieldWeight": "0.5"
    }
  }

• Improved address matching: Address matching has been improved by improving the normalization of the postal code address field.

  Example: 71-75 Shelton street London WC2H9JQ vs. 71 SHELTON STREET LONDON WC2H 9JQ

  • Previously: 0.6239

  • Now: 0.8542

• Improved phone number recognition: Regular expressions for phone number extraction have been improved and now extract more phone number patterns.

• Bug fix: We've partially fixed a problem in Japanese ORG extraction where sometimes the model extracts multiple ORG entities or includes non-related adjacent tokens.

• New per-call option - parameters: Requests can now specify a parameters object to update parameter values for an individual request.  Any non-static parameter can be changed. To specify, add "parameters": {"parameterName":"value"} to the call.

  {
    "name1": {
      "text": "Kraft Services", 
      "entityType": "Organization"
    }, 
    "name2": {
      "text": "Kraft Srvs", 
      "entityType": "Organization"
    }, 
    "parameters": { 
      "deletionScore": "0.2" 
    }
  }

• Expanded English stop words: Organization name matching has been improved by the addition of English stop words.

  Example: SUNY Canton vs. State University of New York at Canton

  • Previously: 0.8340

  • Now: 0.8713

• Improved Spanish, Arabic, and Korean organization matching: By adding the use of word embeddings, we've improved organization name matching when one or both of the names are in the specified language.

  Spanish-English example: Astilleros y Talleres del Noroeste vs. Shipyards and Workshops of the Northwest

  • Previously: 0.4558

  • Now: 0.8838

  Korean-English example: 아시아나 항공 vs. Asiana Airlines

  • Previously: 0.7208

  • Now: 0.8672

  Arabic-English example: الاتحاد العالمي للحفاظ على الطبيعة والمصادر الطبيعية vs. International Union for Conservation of Nature and Natural Resources

  • Previously: 0.3263

  • Now: 0.7106

• Greek coverage expanded: POS tags and lemmas are now recognized for some Greek words previously not identified. (ETROG-3225)

• Bug fix: Fragment detection now counts tokens correctly to determine short lines. This mostly impacts languages without spaces: Chinese, Japanese, and Thai. (ETROG-3177)

• Bug fix: Tokens with digits are now analyzed by for the Greek guesser. (ETROG-3231)

  • Previously: "HDMI1" defaulted to possible PROP, ADJ, NOUN POS tags

  • Now: "HDMI1" gets FM POS tag

• Bug fix: Russian perfective verbs are now lemmatized correctly. Previously some were lemmatized to their imperfective counterparts' lemmas or other incorrect lemmas. (ETROG-3112)

  Example: "разложу" where "разложу" is perfective and its lemma is "разложить". Its imperfective counterpart’s lemma is "раскладывать"

  • Previously: Two analyses: one lemmatized to "раскладывать", the other to "разлагать"

  • Now: One analysis, lemmatized to "разложить"

• Bug fix: German lemmas that consist of a separable prefix and a noun are now correctly capitalized. (ETROG-3235)

  Example: Input "Mitbehandlung"; "mit" is a separable prefix

  • Previously: Lemmatized to "mitBehandlung"

  • Now: Lemmatized to "Mitbehandlung"

• Bug fix: In Hebrew, terminal combining characters are no longer getting split into their own tokens. (ETROG-3224)

  Example: "1" (keycap)

  • Previously: Tokenized to two tokens, <U+0031 DIGIT ONE> <U+20E3 COMBINING ENCLOSING KEYCAP>.

  • Now: Tokenized to one token, "1"

• Arabic improvements:  New PERSON and ORGANIZATION stopwords have been added, improving Arabic-English and Arabic-Arabic name matching.

  Example for PERSON entity type: محمد vs.  نبي محمد

  • Previously: score = 0.4186

  • Now: score = 0.99

  Example for ORGANIZATION entity type: بنك الأهلي التجاري vs. ال البنك الأهلي التجاري

  • Previously: score = 0.6512

  • Now: score = 0.99

• Hebrew - English added: Hebrew to English is now a supported translation language pair. Rosette supports the Hebrew transliteration standards ISO259_2_1994 and ICU (which refers to the default Hebrew transliterator implemented by ICU and is based on the UNGEGN standard for geographic names), but defaults to FOLK. This Basis Technology-created transliteration scheme is more useful than the other more academic standards (ISO259_2_1994 and ICU) as it closely resembles how people in the real world write Hebrew names with Latin characters.

• French added:  The semantic similarity endpoints now support French.

• The activeLearning parameter in the rex-factory-config.yaml file is now being read.

• Dynamic custom profiles Changes to a custom profile can now be loaded without restarting Rosette Enterprise. To dynamically change a custom profile, delete the profile directory from the disk and create a new one with the changes. The new directory can have the same name as the deleted profile directory.

• Support for unfielded addresses:  Addresses no longer have to be provided as separate field components.

  {  
  "address1": "The Book Club 100-106 Leonard St Shoreditch London EC2A 4RH, United Kingdom",  
  "address2": "The Book Club 100-108 Leonard St Shoreditch London EC2A 4RH, UK"
  }"

• Added field overrides: More overrides (matches) have been added for address fields, improving the match scores. Overrides explicitly map nicknames, cognates, and variants to improve matching accuracy.

  Example: England and UK have been added as overrides

  {
  "address1": {    
     "house": "Ffrwdgrech Industrial Estate",    
     "road": "Ffrwdgrech Rd",    
     "city": "Brecon",    
     "country": "UK",    
     "postcode": "LD3 8LA"  
   },  
  "address2": {    
     "house": "Ffrwdgrech Industrial Estate",    
     "road": "Ffrwdgrech Rd",    
     "city": "Brecon",    
     "country": "England",    
     "postcode": "LD3 8LA"  
   }
  }

  • Previously: 0.86

  • Now: 0.95

• Improved support for misfielded address components: We've improved scores by checking for matches in different fields to catch where address components were input into the wrong field.

  Example: Washington and D.C. are matched even though they were put into different fields

  {
  "address1": {    
     "houseNumber": "1600",    
     "road": "Pennsylvania Ave N.W.",    
     "city": "Washington",    
     "state": "D.C.",    
     "postcode": "20500"  
   },  
  "address2": {    
     "houseNumber": "1600",    
     "road": "Pennsylvania Ave N.W.",    
     "city": "D.C.",    
     "state": "Washington",    
     "postcode": "20500"  
   }
  }

  • Previously: 0.72

  • Now: cross-field matching between city and state generates a higher match score of 0.94

• Hebrew improvements: Entity extraction has improved Hebrew normalization. The disambiguator can now identify prefixes removed from the entity's normalized form. Improvements are a result of Hebrew morphology enhancements.

• Bug fix: A new line character in a regex (\n) will now also match carriage returns (\r) and a combination of both (\r\n).

• Hebrew improvements: Hebrew tokens that have prefixes but not stems now get tagged the correct part of speech. Previously, they got the POS tag "unknown".

  Example: “ה” from the string “ה70”

  • Previously: POS tag "unknown"

  • Now: POS tag "quantifier"

• Bug fix: Minimally-qualified emoji are no longer split apart.

  Example: The emoji for "man tipping hand" (<U+1F481, U+200D, U+2642>: )

  • Previously: U+1F481 and <U+200D, U+2642> (2 tokens)

  • Now: <U+1F481, U+200D, U+2642> (1 token)

• Bug fix: Capitalized common nouns are no longer detected as verbs.

  Example: The noun "Service" from the phrase "Price and Quality of Service"

  • Previously: POS tag VI (infinitive or imperative verb)

  • Now: POS tag PROP (proper noun)

• Improved Arabic-English and Arabic-Arabic matching: Many enhancements have improved the name similarity scores when matching Arabic names with Arabic or English names.

  • Token alignment Improved token alignment between names in Arabic and English

    Example: Aung San Suu Kyi vs. أون سان سو تشي

    • Previously: 0.4544

    • Now: 0.7261

  • New statistical model A new Arabic-English statistical model has been added.

    Example: Debby Ryan vs. ديبي رايان

    • Previously: 0.6114

    • Now: 0.9202

  • Gender identification We've added gender identification for Arabic names

    Example: Mario Savinova vs. ماريا سافينوفا

    • Previously: 0.9394

    • Now: مَارِيَا (Maria) gets detected as a female name so a gender correction penalty gets applied and the match score comes down to 0.6413

  • Language model The weighting of tokens in Arabic names has been improved.

    Example: Margaret Thatcher vs. مارجريت تاتشر

    • Previously: 0.6299

    • Now: 0.8182

  • Initials We've added support for initials and initialisms in Arabic.

    Example: J vs. جمال

    • Previously: 0.4106

    • Now: 0.8256

• Japanese-English Name similarity scores for organizations has been improved by expanding the list of organization name equivalents mapped between Japanese-English (aka, token overrides).

  Example: 中国建設銀行 vs. China Construction Bank

  • Previously: 0.8701

  • Now: 1.0

• Windows support: To parse unfielded addresses, address similarity depends on the jpostal binding for the open source libpostal library. Though jpostal is not officially supported on Windows, our tests have shown it to function as expected. Please contact support if you discover any issues.

• More complete sample files: The sample files to build the SQLite connector described in the Custom Knowledge Base Connectors section now includes all files required to build with Maven. The configuration to run the connector with Rosette Enterprise is now provided as well.

• Bug fix: Confidence scores for entity linking now use the same scale, whether linking to Wikidata or a custom knowledge base. Previously, the confidence scores given for links to custom knowledge bases were much lower than those calculated for the Wikidata knowledge base.

• Hebrew improvements: When guessHebrewPrefixes is true, unrecognized Hebrew tokens will now get analyses with and without potential prefixes. Previously, they would only get analyses with potential prefixes.

  Example: Token: "ומפיפרנו"

  • Previously: 2 analyses: 

    hebrewPrefixes=[ו] lemma=מפיפרנו

    hebrewPrefixes=[ו, מ] lemma=פיפרנו

  • Now: 3 analysis:

    hebrewPrefixes=[ו] lemma=מפיפרנו

    hebrewPrefixes=[ו, מ] lemma=פיפרנו

    hebrewPrefixes=[] lemma=ומפיפרנו

• Licensed field: You can now see which languages you are licensed for by endpoint. All /supported-languages endpoints now return a licensed field for each language. The value is true if the you have a license for the language, false if not.

• Hebrew accuracy improvement: We re-trained the statistical model, adding an annotated finance dataset. The accuracy is improved for the finance genre and remains the same for the news genre.

• New DNN model for Hebrew: We added a new deep neural network model for Hebrew. It performs better than the statistical model on both news and finance genres.

• Hebrew normalization: For more accurate downstream processing, we have removed prefixes from normalized Hebrew output, except for the definite article.

• Hebrew proper nouns: We added more proper nouns to the Hebrew lexicon.

• German professions: We added more German professions to the German lexicon.

• Additional emoji support: Emoji hair components are now lemmatized.

• Spanish performance improvements: Spanish disambiguation is now faster.

• Unicode 13.0 emojis: Unicode 13.0 emoji sequences are now tokenized.

• Bug fix: We fixed a bug where low surrogates were stripped from the ends of tokens in Hebrew.

• Known issue: Minimally-qualified emoji zero-width joiner (ZWJ) sequences are no longer recognized as emoji. We will fix this in the next release.

• Offline Docker installation Server shipments now include a helper script, download.sh, to download the components used in an offline Docker installation.

• Geo-coordinates regex: We added supplemental regex support for ISO 6709 geo-coordinates for all languages.

• Linking knowledge bases: You can now load multiple custom knowledge bases and prioritize which knowledge base to try linking with first. Edit the kbs parameter in the rex-factory-config.yaml file.

• Redactor improvement: Redactor weights can now be configured for specific subsources used by processors. For example,  the entity linker can link to multiple knowledge bases, each of which is considered a subsource. These subsources can have their own redactor weights to adjudicate linking conflicts among the subsource.

• Custom profile with custom knowledge base: You can now define a custom knowledge base in a custom profile.

• New license file: Linking to custom knowledge bases is now licensed separately. If you link to one or more custom knowledge bases, you will need to install a new license file. Linking to wikidata does not require a new license file.

• Japanese stopwords: We’ve added the ability to handle Japanese stopwords that contain parentheses.

• Bug fix: We fixed a bug where a multi-digit number containing a space that was adjacent to a symbol was tokenized to multiple tokens in some languages when fstTokenize was enabled.

• Document-only analysis: Entity-level sentiment analysis can now be turned off, allowing document-level sentiment analysis only. See the Rosette Enterprise User Guide for more information.

• Expanded Language support: Users of the Rosette Text Classification Field Training Kit can now train custom sentiment analysis models in any language supported by the tokenization endpoint (for document-level analysis) or the entity extraction and linking endpoint (for entity-level analysis). For more information on the training and configuration procedure, see the Rosette Field Training Kit documentation.

• Version Changes

  Apache Lucene Core updated from v6.6.0_1 to v7.6.0_1 (Apache 2.0 license)

• Bug fix: We've fixed a bug where when the fstTokenize option was enabled, the lemmas of hyphenated Russian compound words only had the the final piece lemmatized. Now both pieces are lemmatized.

  • Previously: человека-волка was lemmatized to человека-волк

  • Now: человека-волка is lemmatized to человек-волк

• New factory configuration option: We’ve added a new factory configuration option, maxResults.  This option can be used to cap the number of results returned.  By default, all results exceeding the score and confidence thresholds (if set) will be returned.

• Korean: We've improved the accuracy of Korean extraction, largely through better handling of Josa (postpositions) and compound words.

• Entity Linking: We've added support for entity linking to Wikipedia for both the top level types (PERSON, LOCATION, ORGANIZATION, ETC.) as well as the over 700 DBpedia types (see full list here) in the remaining 16 languages supported by entity extraction. This is in addition to the languages currently supported by entity linking: Chinese, English, Japanese, and Spanish.

• Hebrew disambiguation: We’ve improved analysis in Hebrew by adding disambiguation, a mechanism for more accurately choosing which of several candidate analyses is provided in the response. For Hebrew only, we've added an option disambiguatorType to select which disambiguator is used. The values are DNN for the TensorFlow-based deep neural network model and dictionary for the dictionary-based model. The default is dictionary. To enable the DNN disambiguator, add {"options": {"disambiguator": "DNN"}} to your call.  

• Persian lemmatization: We’ve added lemmatization support to Persian.

• Chinese organization names: We've improved the accuracy of matching between Chinese and English organization names.

  Previously: The similarity score for 索尔维－恒昌（张家港）精细化工有限公司 and Solvay-Hengchang (Zhangjiagang) Fine Chemicals Co., Ltd was 0.6929.

  Now: The similarity score for 索尔维－恒昌（张家港）精细化工有限公司 and Solvay-Hengchang (Zhangjiagang) Fine Chemicals Co., Ltd score is 0.8225.

• The minimum system requirements for running Rosette Enterprise have changed for some use cases. This is a result of providing entity linking for 16 additional languages in this release. We now support entity linking for all 20 languages supported by entity extraction.

  The entity extraction and linking, sentiment analysis, and topic extraction endpoints require significant memory allocation. If using these endpoints, the new minimum memory requirements are:

  • 32GB RAM

  • 64GB of disk space (more may be needed for growing logs)

  For all other endpoints, the minimum memory requirements are:

  • 16GB RAM

  • 35GB of disk space (more may be needed for growing logs)

• Rosette Enterprise is now available as a Docker container. The images are available on Docker Hub. A Basis shipment, containing a license file and a docker-compose file customized to your licensed endpoints, is still required.

Morphological Analysis /morphology/{morphoFeature} 

• Chinese: We’ve added the word “百度” to the Chinese lexicon. This only has an effect when modelType is set to default, which is its default value.

  Previous:  Two tokens: “百”, “度”

  Now: One token: “百度”

• German: The part of speech of the acronyms “MAN” and “MIT” is now NOUN in German, instead of falling back to the parts of speech of the unrelated words “man” and “mit”.

  Previous: MAN/INDPRO, MIT/PREP, MIT/VPREF, MIT/ADV

  Now: MAN/NOUN, MIT/NOUN

• Spanish: We’ve improved Spanish part-of-speech tag and lemma disambiguation.

  Previous: 91.646% POS accuracy and 90.243% lemma accuracy using the IULA Spanish LSP Treebank

  Now: 91.742% POS accuracy and 90.344% lemma accuracy

• New Model: We’ve added a new model for increased accuracy when matching Hungarian names to other Hungarian names. 

  Previous: Pavlovitch Bryulov vs. Pavlovics Brjullov - score: 0.84

  Now: Pavlovitch Bryulov vs. Pavlovics Brjullov - score: 0.95

• Bug Fix: Fixed a bug involving duplicate readings produced when transliterating Chinese names.

  Previous: Wang Xing vs. 王行 - score: 0.69

  Now: Wang Xing vs. 王行 - score: 0.99

• Improved Chinese accuracy: We've replaced the underlying tokenizer to improve accuracy in Chinese.

• Improved Hungarian accuracy: We've updated our pattern match extractors in Hungarian. This improves accuracy for the MONEY and DATE types.

• Known issue: We've fixed a bug where entity mentions were being miscounted if the calculateSalience option was set to true.

• Improved Dutch disambiguation: We've improved Dutch part-of-speech disambiguation.

• Improved Japanese and Chinese lemma support: In the last release, most Japanese and Chinese tokens did not have lemmas when modelType was set to default. Now, such tokens have lemmas equivalent to their surface forms.

• New per-endpoint licensing: Endpoints are now activated directly from your installed license. The endpoints.yaml file has been removed from the installation.

• New Enterprise User guide: We’ve added a user guide (rosette-enterprise-user-guide-1.11.0.pdf), that provides new content and  replaces the files rosette-api-on-premise-install-guide-1.11.0.txt, overview.md, and Rosette_API_Embedded_User_Guide.pdf.

• Simplified installation for macOS and Linux: The installation for both RESTful and embedded Java has been simplified for macOS and Linux. Installation for Windows has not changed, but detailed installation notes are now included as part of the new Enterprise User Guide.

• Name Changes: We are continuing to consolidate and simplify our branding. Rosette API On-Premise is now Rosette Enterprise. We’ve made changes to the documentation and license names to reflect.

• New supported languages sub-endpoints: For all endpoints (excluding Name Similarity, Name Translation, and Name Deduplication), Rosette now provides a GET /rest/v1/<endpoint>/supported-languages method that returns that endpoint's supported languages and scripts. See the Features and Functions or the Interactive Docs for more information.

• Updated bindings: We've updated our CSharp and Java bindings. Be sure to get the latest version (1.11.0) to take advantage of all the new features and improvements!

• New language: Rosette now supports name deduplication of Hungarian names.

• Multilabel categorization: Rosette can now return multiple category labels per document.  For more information, see the Features and Functions. To return only a single category label per document, set the {"options": {"singleLabel": true}}. For more information, see the Features and Functions.

• New languages: The text embedding endpoint now supports Russian, North Korean, South Korean, and Arabic.

• Individual token embeddings: We can now return embeddings for individual input tokens. To enable per-token embeddings, add {"options": {"perToken": true}} to your call.

• Response modifications: We've made changes to the text embeddings endpoint's response structure. Document-level embeddings now have their own  dedicated slot embeddings and will no longer appear in documentMetadata. Please note that this is a breaking change, contact Rosette support for more information.

• New feature - DBpedia Types (LABS): We've added over 700 new entity types to the Entity Extraction and Linking endpoint, drawn from the DBpedia ontology. To access these entity types, add {"options": {"includeDBpediaType" = true}} to your call. You'll notice more than 10 additional macro types in the type field as well as the all new DBpediaType field. For more information, see the Features and Functions. Note that this feature is still in LABS and subject to change. Send us your thoughts!

• Better accuracy: We've improved the recall of Rosette's entity linking across all supported languages.

• New language: Entity extraction now supports Hungarian.

• Known issue: MONEY, PHONE NUMBER, and URL types are not extracting properly in Hungarian. This will be fixed in the September 2018 patch release.

• Support for North and South Korean added: Rosette can now identify North Korean (qkp) and South Korean (qkr) dialects. To enable the dialects, add { "options": { "koreanDialects":true }} to your call.

• New algorithm for Chinese and Japanese: We've added a new algorithm for Chinese and Japanese morphological analysis. Prior to version 1.11.0, the default algorithm was perceptron. To return to the old model, add {"options": {"modelType": "perceptron"}} to the body of your call.

• Norwegian lemmatization: We’ve expanded the lemma dictionaries for Norwegian, both Bokmål and Nynorsk.

• Improved English and Spanish disambiguation We've improved the accuracy of lemmatization and part of speech tagging in both English and Spanish.

• Bug fix: We've improved handling of formatting characters in German. 

• Bug fix: We’ve fixed a bug where the Hebrew POS tag wPrefix was not converted to UPT-16.

• New algorithm for Chinese and Japanese: We've added a new algorithm for Chinese and Japanese tokenization. Prior to version 1.11.0, the default algorithm was perceptron. To return to the old model, add {"options": {"modelType": "perceptron"}} to the body of your call.

• Bug fix: We’ve fixed a bug where in Catalan, in which Rosette did not tokenize after an apostrophe in cases where the apostrophe marks a token boundary.

• Bug fix: We've fixed a big in Japanese, in which Rosette did not recognize 々 as a Japanese character, so it was considered its own token.

• New language: Name similarity now supports matching between Hungarian and English names.

• Improved language-of-origin detection: We've improved the detection of language-of-origin of Japanese names written in Katakana.

• Improved Arabic script segmentation: We've improved segmentation of Persian (Dari and Farsi), Pushto, and Urdu names written in Arabic script.

• New Japanese tokenizer: We've replaced the underlying tokenizer to improve accuracy for Japanese.

• Bug fix: We've fixed a bug where some components of compound German words were incorrect when the surface form of the component could be either a noun or a verb. 

• Bug fix: We've fixed a bug where the Hebrew parts of speech tags did not use UPT-16, the POS tag set used by the other languages.

• Bug fix: We've fixed a bug where returned email addresses and URLs could contain control or whitespace characters.

• Bug fix: We've fixed a bug where returned Hebrew tokens could contain control characters or nothing but default ignorable characters.

• Bug fix: We've fixed a bug where Chinese, Japanese, and Thai tokens could contain control characters.

• Bug fix: We've fixed a bug where the default Japanese tokenizer truncated some katakana tokens when they appeared after non-katakana tokens.

• Name Deduplication input limit: Each call now has a limit of 1000 names per list.

• Improved Entity Level Sentiment Analysis: We've improved our calculation of entity level sentiment to more accurately consider the context around each mention. Please note, this update may cause results to change.

• Bug fix: We've fixed a bug where the initial token dependency for every sentence (other than the first) was omitted from the list of results.

• Improved Hebrew Entity Extraction: We've improved Hebrew entity extraction by removing superfluous prefixes from extracted entities.

• Improved Confidence Scores: We've improved statistical model confidence scores to provide a more effective tradeoff between precision and recall. Please note, this update may cause results to change. If you have set a threshold based on entity confidence scores, please evaluate to ensure optimal performance.

• Improved Entity Normalization: Social media characters such as "@" and "#" are removed from a Mentions normalized string. Offsets to the original string data field remain the same.

• Bug fix: Previously, the Dutch disambiguator would always choose analyses whose lemmas matched their surface forms, even for very rare lemmas; now the more common lemma will be returned. For example, schepen can be a singular noun with the lemma schepen, but it is more likely to be a plural noun with the lemma schip.

• New deep neural network processor (in BETA): We've added an alternative entity extraction processor, which can be used in place of the standard statistical extractor. The new processor employs a deep neural network that improves accuracy up to 7% and error rate up to 32%. It is available for English, Arabic, and Korean. To enable this processor, provide DNN for the modelType. Example: {"content": "your_text_here", "options": {"modelType": "DNN"}}

• New language support: We've added support for lemmatizing Catalan, Estonian, Serbian, and Slovak text.

• Bug fix: Previously, tokens could be empty or contain only invisible characters. Such tokens will no longer be returned.

• New language support: Short string language identification now also supports Malay and Indonesian. Both languages were already supported for longer texts.

• New language support: We've added support for sentiment analysis in Persian (Farsi and Dari) at both the document and the entity level.

• New language support: Rosette now supports transliteration of person, organization, and location names from Greek to Latin script.

• New language support: Rosette now supports matching of Greek names written in Greek script to English names written in Latin script and other Greek names written in Greek script.

• Accuracy improvements: We have improved match scores and segmentation rules for Arabic, Western Farsi, and Japanese names.

• Bug fix: We’ve improved our handling of tokens consisting of numbers and Latin characters, such as serial numbers, in Korean. Previously these tokens were decompounded into multiple morphemes.

• Bug fix: We’ve added the lower case Russian word “интернет” (“internet”) to the dictionary, which was previously only present in title case.

• Bug fix: We’ve improved our handling of tokens containing an apostrophe immediately followed by a digit in languages like French and Italian, like “all'M5S”. Previously, the apostrophe would be parsed as its own token.

• Bug fix: We’ve improved our German analysis by taking better advantage of context clues, and now return more accurate results, especially for uncommon words.

• Bug fix: We’ve improved our handling of English and German words in all-caps. Previously, these words were assumed to be proper nouns, even though all-caps may simply denote emphasis.

• Performance improvement: We’ve added caching of model objects to prevent OutOfMemoryErrors.

• Hebrew entity extraction: We’ve added support for entity type "Title" in Hebrew.

• German disambiguation: We’ve improved our German disambiguator for lemmas and part-of-speech tags to be more sensitive to capitalization, particularly for single word inputs.

• Bug fix: Previously, German definite articles (der, die, das, den, dem, and des) meaning the were lemmatized inconsistently. They are now all lemmatized to the masculine singular nominative form, der.

• Bug fix: In some languages, an apostrophe may mark a token boundary, like in the Italian phrase all'M5S. Previously the token boundary was incorrectly omitted when the following token contained a digit. This issue has been rectified and M5S will be properly tokenized.

• Farsi Name Matching: We’ve improved the behavior of Western Farsi-English matching by tokenizing input names earlier in the analysis process.

• Currency support: We’ve added several additional currency symbols to the regex, including the Turkish Lira (₺), the Pound Sterling (₤), and the Euro (€).

• Bug fix: We’ve fixed a bug that caused hexadecimal number strings to be incorrectly extracted as products.

• Thai improvement: We’ve modified the gemination rules (consonant elongation) of the ISO11940-2 Thai transliteration standard to improve Thai translation accuracy.

• Bug fix: We’ve fixed a bug around matching names of organizations and locations that contain numbers, such as "Century 21 Real Estate LLC." These non-person names containing digits will now match more accurately.

• Salience scores: We've added salience scores for keyphrases and concepts to indicate how relevant an extracted concept or keyphrase is to the overall content of a text. You now have the option to filter out results below a desired threshold value: {"content": "your_text_here", "options": {"keyphraseSalienceThreshold": value, "conceptSalienceThreshold": other_value}}. 

• Short string support: We've improved our concept extraction logic, and now support concept extraction for short input strings, i.e. texts less than 280 characters long.

• Thai support: Rosette now supports deduplication of Thai names.

• New feature: We've added the option to use an experimental alternative deep neural network (DNN) sentiment model for English: {"content": "your_text_here", "options": {"modelType": "DNN"}}. The new model will produce different results, which may be more accurate than the current support vector machine (SVM) model, depending on your data. As it is experimental, we are particularly interested in getting user feedback. On-premise users of Rosette API should review the new system requirements in install-guide.txt before using this option.

• Entity offsets returned: Entity mention offsets are now returned by default. Offsets can be used to locate the exact surface forms of an extracted entity in the document text.

• Korean improvements: We’ve significantly improved the accuracy of entity extraction results across all entity types in Korean.

• Confidence scores: Confidence scores for entities extracted using Rosette’s statistical processor, as well as all linked entities, will now be returned by default. Confidence scores allow Rosette to return the most accurate results, particularly for entity linking. To change this behavior, set {"content": "your_text_here", "options": {"calculateConfidence": false}}.

• Detect language regions in multilingual documents: The language identification endpoint can now detect different language regions in a multilingual document.

• Score changes: We’ve rescaled the confidence scores returned by the language identification endpoint based on customer feedback. The ranking of language candidates will not change, but the scores themselves will be higher. If you currently filter language identification results based on a confidence threshold, you will need to reset that threshold to maintain parity with previous versions.

• Thai support: Rosette now supports transliteration of names from Thai to Latin script.

• Thai support: Rosette now supports matching of Thai names to English names and other Thai names.

• Accuracy improvements: We have improved match scores for Arabic names (persons, locations and organizations) as well as for Chinese and Japanese organizations.

• Bug Fix: This release addresses a bug whereby entity linking confidence scores were not being returned when requested. Confidence scores for entities resolved to Wikipedia entries will now be returned when using the following option: {"content": "your_text_here", "options": {"calculateConfidence": true}}

• New Endpoint: Topic extraction We've added a topic extraction endpoint that identifies the key ideas of an input text. For a given input, the endpoint will return two lists: Keyphrases, a list of phrases extracted directly from the text, and Concepts, a list of phrases which do not have to be explicitly mentioned in the input.

• LABS graduates: The /transliteration, /relationships, and /syntax/dependencies endpoints have graduated from “Labs” status and are now fully supported.

• New language: Rosette now supports document and entity-level sentiment analysis in French.

• Salience Scoring: Rosette can now return salience scores, which indicate whether an entity is important to the overall scope of the document. Turn on the scores by adding an option to the request: {"content": "your_text_here", "options": {"calculateSalience": true}}

• Linking Confidence Scoring: Rosette can also now return Linking Confidence scores, which represent the degree of certainty of the link between an in-document entity mention and its linked QID. It may be used for thresholding and removal of false positives. Linking Confidence scores for entities identified by our linker and assigned with a QID are now available by adding an option to the request: {"content": "your_text_here", "options": {"calculateConfidence": true}}

• Bug fix: This release addresses a backward compatibility issue between the latest Rosette API and older versions of our Java binding that affected Rosette's ability to return entity confidence scores. Confidence scores for entities identified by our statistical extractor are now available by adding an option to the request: {"content": "your_text_here", "options": {"calculateConfidence": true}}

• New Endpoint: Transliteration We've added a transliteration endpoint that converts between Arabic written in ASCII, also called Romanized Arabic chat or Arabizi, and native Arabic script.

• Beta Arabic Support for /sentiment: We now return document-level and entity-level sentiment analysis results for Arabic language input.

• Personal pronoun resolution for /relationships: Building on the pronoun resolution capabilities of our /entities endpoint, pronouns which are resolved to named entities can now be arguments in relationships.

• Improved Confidence Scoring: Confidence score calculation is improved to correlate well with precision and may be used for thresholding and removal of false positives.

• New support for emoticons, emoji, @mentions, hashtags, URLs, and email addresses: These special characters and character combinations are now kept together as a single token in all languages, greatly improving the accuracy of analysis further downstream.

• Improved accuracy for English and Spanish: For this release, we updated our English and Spanish dictionaries. We also introduced new, more advanced disambiguation models for these languages, which help Rosette to correctly determine a given word’s part of speech. For example, words like “object” can be either a noun (“this is an object”) or a verb (“I object!”).

• Lemmatization and normalization of emoticons, emoji, @mentions, hashtags, URLs, and email addresses: Rosette now normalizes and lemmatizes these special characters and character combinations to streamline analysis.

• Improved decompounding for Dutch: Dutch language text is now decompounded more accurately, Dutch text is now decompounded more accurately, producing better tokens for search enhancement and other applications.

• Improved Accuracy of Corporate Relationships: Improvements made to the identification of relationships between corporations. The relationships involved are: ORG-SUBSIDIARY-OF, ORG-COLLABORATORS, ORG-ACQUIRED-BY and ORG-PROVIDER-TO.

• Removed the ORG-PARTNERSHIPS Relationship: The ORG-PARTNERSHIPS relationship is now subsumed under ORG-COLLABORATORS and is no longer extracted as an independent relationship.

• Improved Linking Accuracy via Inclusion of New Context Features: The statistical model for entity linking includes features that measure the vector space similarity between an entity context and the Wikipedia contexts of its potential linking targets. The new features result in higher F-Scores across all supported languages.

• Entity Linking in Japanese, Chinese and Spanish: Entity linking to Wikidata with QIDs for Japanese, Chinese and Spanish text is supported.

• Removed Long Text Linking: Entity linking to Wikidata (with QIDs) for long texts is removed, which, as a result, removed entity linking capabilities in Arabic.

• Vector dimension reduced from 512 to 300: We are able to produce smaller vectors that are more efficient and memory friendly without sacrificing overall speed or accuracy.

• Improved Speed and Accuracy: A number of speed enhancements have been made along with much larger vocabularies to increase accuracy.

• Improved Accuracy on Texts with Mixed Scripts: A script specific model is now selected based on the weighted frequency of the different scripts in the input.

• Japanese Improvements: Rosette API now has better support for Japanese name matching. This includes the new use of word embeddings, which are used to match words with similar semantic meaning, as well as improved Japanese name segmentation.

• New Endpoint Functionality: The /relationships endpoint now returns targeted relationships, as opposed to the former open relationships, as its default extracted relationships. Targeted relationships are specifically between two entities, and are labeled by a certain relationship type. You can see the former open relationships by setting the option of "discoveryMode" to "true".

• Removed Deprecated Endpoint: The /entities/linked endpoint, previously deprecated, is now completely removed. All functionality is available through the /entities endpoint. You will receive a 404 when calling /entities/linked.

• Social Media Linking in Japanese and Chinese: Our fast short text entity linker to Wikidata is now available for Japanese and Chinese.

• Removal of long text entity linking: Our long text entity linker has been replaced by our fast short string entity linker. You will now see entity linking results from our short string linker by default. This removes linking support for Arabic.

• Additional Language Support: The entity extractor now supports Vietnamese.

• Name Translation and Similarity CJK Improvements: The /name-similarity and /name-translation endpoints now support matching and translating between Japanese-Chinese, Japanese-Korean, and Korean-Chinese. Japanese accuracy was improved significantly.

• Improved Accuracy for Document-level Embeddings: We made some improvements to our algorithm for calculating text embeddings across multi-word input, so you should see more accurate results for document-level vectors.

• Beta Japanese Support for /sentiment: We now return document-level and entity-level sentiment analysis results for Japanese language input.

• New Endpoint: We've added a syntactic dependencies endpoint that returns the parse tree of the input text as a list of labeled directed links between tokens, as well as the list of tokens in the input sentence.

• Entities Linked to Wikidata: Where available, Rosette will now link entities extracted within relationships to Wikidata. You'll see this information returned as a QID in the argument ID.

• Modality Returned: We've also added a "modality" field to Rosette's Relationship Extraction. Modality is the semantic context of the possibility or necessity of the relationship; the values can be “assertion”, “negation”, “uncertainty”, “opinion”, or “question”.

• New $99 API Plan: For a limited time, we’re offering a special Starter plan. $99/month gets you 40,000 Rosette API calls. Want to dive deep into Rosette but don’t need a whole 100,000 calls? This plan is for you.

• New Endpoint: We added a text embedding endpoint that returns a single vector of floating point numbers that represents the document or word in the semantic space.

• Additional entities: We changed the /sentiment endpoint to return the sentiment of all entities discovered by Rosette, including Person, Location, Organization, Date, Time, and more entity types.

• Turn off entity linking: We added an option to disable entity linking in order to improve the call speed. Add "options": {"linkEntities": "false"} to your /entities call. Rosette returns a list of the entities with a temporary ID (TID).

• Concurrency header: We added the X-RosetteAPI-Concurrency header to return the number of concurrent calls allowed on your plan. If you are receiving 429 errors, Too Many Requests, then Contact us for greater concurrency.

• Input genre: The genre field is available for /entities and /entities/linked to indicate the input is from social media. Specifying genre=social-media does not affect the output of the other endpoints. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences.

• Temporary entity ID: With the unification of the /entities/linked and /entities endpoints, the /entities/linked now returns a “T” ID for entities without a Wikidata QID.

• Entity endpoints unified: We combined the /entities and /entities/linked endpoints into one endpoint, /entities. Rosette now returns the entity mentions and the entityId, if available. The entityId replaced the indocChainId. The output of /sentiment has not changed.

• Entities Linked deprecated: We deprecated the /entities/linked endpoint. It is still available, but we recommend that you adapt your applications to the new /entities endpoint.

• Additional entities: Rosette now extracts more entity types: Date, Time, Longitude and Latitude, and Distance.

• Japanese entityId: We added support for linking entities in Japanese (jpn) text to their entityId.

• Spanish social media: We added support for extracting entities from social-media in Spanish language documents, using the genre field.

• Malay entities: We added support for extracting entities in Malay (msa).

• 409 Error: We added the 409 error code for when the binding version is out of date. If you receive this error, update your binding to the most recent version.

• Spanish support: We added support for analyzing the sentiment of Spanish language documents.

• Ruby: We added the Ruby binding to the gray column to the right and on Github. There is a Ruby gem available as well.

• R: We added the R binding to the gray column to the right and on Github.

• cURL examples: We changed the shell examples in the gray column on Features and Functions to be cURL code examples.

• Social input: We added a request field, "genre": "social-media", to speed up and improve the accuracy of linking Person, Location, Organization and Product entities in social media posts. English input only.

• Language used: We added a Response Header, X-RosetteAPI-ProcessedLanguage, to return the language used by Rosette for processing the call. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences

• requestId moved: We moved the requestId object from the JSON response body to the Response Header as X-RosetteAPI-Request-Id. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences, /name-translation, /name-similarity

• Rosette API Key: We changed the user_key header’s name to X-RosetteAPI-Key. The user_key header is deprecated. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences, /name-translation, /name-similarity

• unit parameter removed: We removed the optional unit request parameter. All input will be handled as a doc. Applies to: /entities, /entities/linked, /relationships, /sentiment, /morphology

• Base64: We removed support for Base64 encoding. You can submit binary files as a multipart/form-data call type. Applies to: /entities, /entities/linked, /relationships, /categories, /sentiment, /language, /morphology, /tokens, /sentences

• Confidence removed: The confidence value has been removed from the response object.

• Accuracy mode: We removed the optional accuracy mode. All input will be processed with the precision accuracy mode, so Rosette will return a precise list of accurate relationships.

• Explanations removed: The explanations value has been removed from the response object.

• Explanations removed: The explanations value has been removed from the response object.

• Entity sentiment: We added support for entity-level sentiment analysis. The JSON response for the /sentiment endpoint now includes two objects – document and entities. See the interactive documentation for examples of this new response.

• Neutral result: We added a neutral label for documents and entities with a neutral sentiment.

• Short strings: Rosette will automatically process short and long content with our proprietary algorithm for sentiment analysis.

• Explanations removed: The explanations value has been removed from the response object.

• Added language support: We added language support for Dari, Persian, Urdu, and Western Farsi for Parts-of-Speech Tags.

• Universal POS Tags: We return Universal Parts-of-Speech Tags for all supported languages.

• Tokens list: Rosette returns parallel lists of tokens, lemmas, compound components, parts-of-speech tags, and Han-readings. If a token does not have a lemma, compound component, POS tag, or Han-reading, or if the language is not supported, then Rosette will return “null” in that list.

• Renamed to /name-translation: To clarify the endpoint’s function, we renamed /translated-name to /name-translation. /translated-name is no longer available.

• Removed result layer: Within the response to /name-translation and /name-similarity endpoints, we removed the result layer so the results are in the response object. Also applies to: /name-similarity

• TargetScheme requires uppercase: For advanced users who would like to specify a targetScheme, the scheme must be submitted in uppercase.

• Renamed to /name-similarity: To clarify the endpoint’s function, we renamed /matched-name to /name-similarity. /matched-name is no longer available.