リリースノート

[en] Expanded Chinese lexicon: We've expanded the lexicon of multi-character Chinese surnames when tokenizerType is set to spaceless_lexical. (ETROG-3616)

[en] Expanded Japanese lexicon: We have expanded the Japanese lexicon that is used when tokenizerType is set to spaceless_lexical. (ETROG-3632)

[en] Added secondary parts of speech: We've added support for secondary parts of speech to Chinese and Japanese when tokenizerType is set to spaceless_lexical. (ETROG-3636)

[en] Improved support for Chinese readings when tokenizerType is set to spaceless_lexical:

[en] Solr and Lucene support: Lucene 9.5 - 9.7 and Solr 9.3 are now supported (ETROG-3643)

[en] We fixed a bug where an ArrayIndexOutOfBoundsException occurred when the Chinese dictionaries produced more than 6 matches and tokenizerType was set to spaceless_lexical. (ETROG-3635)

[en] When Chinese readings are constructed by character and tokenizerType is set to spaceless_lexical, an apostrophe is now inserted before pinyin syllables that start with "a", "e", or "o" which are not the first syllable. (ETROG-3637)

[en] We fixed a bug where the UPT-16 conversion where some Japanese particles part of speech were not tagged correctly. The particles are now tagged correctly as ADP. (ETROG-3526)

[en] The plugin for Solr 8.11 may throw an exception when using multiple neural models.

[en] Lucene and Solr support: RBL-JE now supports Lucene 9.2 to 9.4 and Solr 9.2. (ETROG-3631)

[en] We fixed a bug where processing an empty string as Chinese or Japanese with tokenizerType set to spaceless_lexical would throw a NullPointerException. (ETROG-3629)

[en] In Solr 8.11.2, it is not possible to consistently load more than one neural model at a time. (ETROG-3608)

[en] We've improved the time it takes to tokenize extremely long (> 10K characters) Japanese sentences. (ETROG-3602)

[en] In Solr 8.11.2, it is not possible to consistently load more than one neural model at a time. (ETROG-3608)

[en] Annotating Ukrainian with universalPosTags set to true no longer results in a RosetteUnsupportedLanguageException being thrown. (ETROG-3604)

[en] Ukrainian support added: Tokenization, sentence boundary detection, segmentation user dictionaries, and many-to-one normalization dictionaries are supported for Ukrainian. (ETROG-3594)

[en] Improved part of speech tags: Language-neutral tokens (numbers, symbols, and punctuation) now get part of speech tags in Indonesian, Standard Malay, and Tagalog. (ETROG-3574)

[en] GPU support: Features that use TensorFlow now use a GPU if available. (ETROG-3564)

[en] Emoji support: Emoji 15.0 is now supported. (ETROG-3577)

[en] New option for Katakana: We've added the option joinKatakanaNextToMiddleDot to control whether sequences of Japanese Katakana tokens adjacent to a middle dot should be merged into a single Katakana token. By default, it is true, which matches the behavior in previous versions of RBL-JE. (ETROG-3592)

[en] Solr 9.1 support: Lucene and Solr 9.1 are supported. (ETROG-3597)

[en] The Japanese POS tag VN (verbal noun) is now mapped to the UPT-16 POS tag NOUN. It was previously mapped to VERB. (ETROG-3583)

[en] Tagalog support:

[en] Indonesian (ind) support: RBL now supports lemmatization for Indonesian, which is the standardized form of Malay spoken in Indonesia. (ETROG-3563)

[en] Standard Malay (zsm) support: RBL now supports lemmatization for Standard Malay, the standardized form of Malay spoken in Malaysia. (ETROG-3563)

[en] We fixed a bug in Russian where certain uncommon consonant–vowel sequences in words in the lexicon were incorrectly replaced with more common sequences with different vowel letters. (ETROG-3541)

[en] Example: брошюра

[en] An open source package (JavaCPP) has been updated to allow the Elasticsearch plugin to use TensorFlow. (ESPI-168)

[en] RBL-JE no longer crashes when loading TensorFlow fails with a NoClassDefFoundError. It now throws an exception. (ESPI-169)

[en] Indonesian support added: RBL now supports Part of Speech (POS) tagging in Indonesian. (ETROG-3543)

[en] Malay (Standard) support added: RBL now supports Part of Speech (POS) tagging in Malay (Standard). (ETROG-3545)

[en] Russian lexicon improved: We've added many words related to computer technology to the Russian lexicon. (ETROG-3523, ETROG-3538)

[en] Java 17 support added: Java 8 and 9 support has been removed. (ETROG-3524)

[en] Solr 9 support added: RBL now supports Lucene and Solr 9. (ETROG-3549)

[en] Solr 6 support deprecated: RBL no longer supports Lucene or Solr 6 or earlier. (ETROG-3519)

[en] In Japanese, negative forms of ichidan verbs written all in hiragana are no longer lemmatized to end with “なう”. (ETROG-3534)

[en] Example: Input: くれない

[en] Solr 8.11 support: This release supports Solr 8.11 (ETROG-3502)

[en] Deprecated methods: Token#getType has been deprecated as token types are not used in RBL-JE without the Lucene/Solr plugins and the plugins use a different API. (ETROG-3503)

[en] Solr 6 support deprecated: Support for Solr versions 6.x and earlier is deprecated as of this release and will be removed in the next version.

[en] Permission changes: We removed group and other write permissions from model files. All files are now only writable by the owner. (ETROG-3516)

[en] RBL no longer crashes in Arabic when emoticons is enabled. This fixes a bug introduced in 7.42.1. (ETROG-3493)

[en] In Korean, emoji and other language-neutral tokens no longer cause a ClassCastException to be thrown when using the ADM API. (ETROG-3488)

[en] Deprecated factories: TokenizerFactory, AnalyzerFactory, and CSCAnalyzerFactory have been deprecated in favor of BaseLinguisticsFactory. (ETROG-3453)

[en] Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts, for Japanese with tokenizerType set to spaceless_lexical. (ETROG-3474)

[en] Example: Input: 三・一四

[en] Emojis: U+3030 and U+303D are now tagged as emojis even when not followed by U+FE0F. (ETROG-3478)

[en] Emoji support: We now support the emoji in Unicode 14.0 (ETROG-3476)

[en] Japanese tokenization: In Japanese, when tokenizerType is set to spaceless_lexical, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. This is consistent with the default algorithm, spaceless_statistical. (ETROG-3475)

[en] Solr 8.10 support: This release supports Solr 8.10. (ETROG-3482)

[en] Improved POS tags: Many number, punctuation, and symbol characters are now POS-tagged appropriately as numbers, punctuations, and symbols instead of being marked as unknown or some other tag. This applies to all languages with POS tags. (ETROG-3481)

[en] Hungarian improvements: We've added some Hungarian abbreviations and improved sentence boundary detection around Hungarian abbreviations. (ETROG-3479, ETROG-3484)

[en] In Japanese, when tokenizerType is set to spaceless_lexical, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)

[en] We've reverted two of the POS changes made in version 7.39.0.63.0 as they introduced regressions in Chinese and Japanese. (ETROG-3466)

[en] The values are now:

[en] RBL-JE no longer detects characters as emoji when followed by the text presentation selector (U+FE0E). (ETROG-3480)

[en] In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)

[en] Enabling universalPosTags for Indonesian, Tagalog, or Standard Malay no longer throws a RosetteUnsupportedLanguageException. POS tags are not supported for these languages, so the universalPosTags option is ignored. (ETROG-3465)

[en] New language support: Tokenization is now supported for Indonesian, Standard Malay, and Tagalog. (ETROG-3443)

[en] Fragment definition: A single line followed by an empty line is no longer always considered a fragment. They are still considered fragments if the line is short, as specified by the maxTokensForShortLine parameter. (ETROG-3431)

[en] Solr 8.9 support: This release supports Solr 8.9. (ETROG-3457)

[en] We fixed a bug where enabling alternativeSpanishDisambiguation for Spanish caused a NullPointerException to be thrown. (ETROG-3435)

[en] We fixed a bug where setting disambiguatorType to DNN for Hebrew caused a RosetteRuntimeException to be thrown. (ETROG-3437)

[en] New option for tokenizers: We've added a new option, tokenizerType to specify which tokenizer to use. The options alternativeTokenization and fstTokenize are deprecated in favor of tokenizerType. (ETROG-3419)

[en] New Korean tokenizer: We've added a new tokenizer for spaceless Korean input. The previous tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. Activate it by setting tokenizerType to spaceless_statistical. (ETROG-3392)

[en] Multiple punctuation characters are no longer returned as a single token in Chinese when alternativeTokenization is true or tokenizerType is set to spaceless_lexical. Now each character is its own token. (ETROG-3402)

[en] Example: Input: 天津？？

[en] Improved Hebrew tokenizer and new analyzer: The Hebrew tokenizer is now more consistent with the tokenizers of other languages. Hebrew tokenization and analysis are now done in separate steps.(ETROG-3290)

[en] Normalized characters: Normalized half-width and full-width characters are processed the same as their counterparts. (ETROG-3351)

[en] Solr 8.8: This release supports Solr 8.8. (ETROG-3369)

[en] Improved CSCAnnotator output: The CSCAnnotator now emits tokens in addition to translations, even if no tokens were specified in the input. (ETROG-3356)

[en] Improved directory structure: The contents of the models/ directory are now separated into subdirectories by language. (ETROG-1218)

[en] Statistical models moved to models/ directory: The following files have been moved from dicts/ to models/: (ETROG-1218)

[en] Hebrew tokens containing a geresh are now tokenized properly. Previously, only the part up to the geresh would be returned as the token text, and the part after the geresh would sometimes be considered a suffix. Now the whole token is returned as the token's text. (ETROG-3262, ETROG-3290)

[en] Example: מע'רב

[en] A structured region containing two new lines is now properly labeled as STRUCTURED. Previously, the layout region would be labeled as UNSTRUCTURED. (ETROG-3378)

[en] Example: * item\n* item\n* item\n\n

[en] RBL-JE no longer normalizes certain emoji ZWJ sequences to U+1F48F KISS, U+1F491 COUPLE WITH HEART, and U+1F46A FAMILY, to be consistent with Unicode’s efforts to make emoji more gender-neutral by default. (ETROG-3350)

[en] We reverted some changes to Korean disambiguation from the 7.37.0.c62.2 release as the changes introduced new disambiguation errors. (ETROG-3349)

[en] Greek lexicon: The Greek lexicon has additional words. (ETROG-3288)

[en] Greek disambiguation improved: Certain Greek forms are now disambiguated to prefer a modern analysis over an archaic analysis. alternativeGreekDisambiguation must be set to false, which is the default. (ETROG-3289)

[en] Example: δείξε

[en] New Greek disambiguator added: The new Greek disambiguator is more accurate, but slower. The new disambiguator is enabled by default. To use the old disambiguator, set alternativeGreekDisambiguation to true. (ETROG-3304)

[en] Deprecated classes: The classes BufferWordBreaker and WordBreakResults have been deprecated. (ETROG-3318)

[en] The Greek guesser now handles tokens with non-alphanumeric characters. (ETROG-3286)

[en] Example: Start+

[en] GenericTokenizer#hasNext is now implemented to be consistent with the documentation for Iterator#hasNext. Previously it always returned false. (ETROG-2140)

[en] Performance improvement: Spanish disambiguation with alternativeSpanishDisambiguation set to false is now faster. (ETROG-3271)

[en] Performance improvement: Korean disambiguation is now faster. (ETROG-3280, ETROG-3282)

[en] Support for unknown language: If the language is unknown (xxx), tokenization and sentence breaking is supported. (ETROG-3278)

[en] Solr 8.7.0: We now support Solr 8.7.0. (ETROG-3315)

[en] Tokenization rule preprocessor: The preprocessor command !!btinclude is supported in tokenization rule files, supporting inclusion of files in rule files. (ETROG-2497)

[en] Updated sample: The tokenize-analyze sample has been changed from two applications running in sequence to a single application that both tokenizes and analyzes. (ETROG-3291)

[en] New sample: The sample csc-annotate demonstrates using CSC with the ADM API. (ETROG-3317)

[en] Deprecated option: TokenizerOption#includeRoots has been deprecated and replaced with TokenizerOption#includeHebrewRoots. (ETROG-3314)

[en] Deprecated option: The alternative tokenization option deliverExtendedAttributes is now deprecated. Previously it delivered an unsupported extended property. (ETROG-3311)

[en] Combining characters in Hebrew which were being erroneously split into tokens separate from their bases are now not being split. (ETROG-3277)

[en] A clear exception (RosetteUnsupportedLanguageException) is now thrown when tokenizing some unsupported languages. Previously, these languages appeared to work. The same tokenizer is still available by specifying the unknown language (xxx). The languages impacted are Albanian, Bulgarian, Croatian, Indonesian, Malay, Slovenian, Standard Malay, and Ukrainian. (ETROG-3278, ETROG-3326)

[en] RBL no longer crashes when alternativeTokenization and fragmentBoundaryDetection are both enabled for some inputs in Japanese and Chinese. (ETROG-3285)

[en] Correct start and end offsets are now produced when fstTokenize is set to true. Previously, some Spanish inputs would produce tokens with start and end offsets of 0. (ETROG-3292)

[en] The mappings of default Basis POS tags to universal POS tags (UPT-16) have been corrected for Greek. (ETROG-3306)

[en] Tokens no longer have null token types. (ETROG-3316)

[en] When an NFKC normalized character results in multiple tokens, those tokens no longer have equal start and end offsets. Previously this could occur when nfkcNormalize was set to true. (ETROG-2505)

[en] Example: ﷺ

[en] Lucene/Solr: Versions up through 8.6.0 are now supported. (ETROG-3250)

[en] Decompose compounds: The option to control decomposition of compounds is now available in Dutch, German, Hungarian, Danish, Bokmål, Nynorsk, Swedish, and Korean. The default for decomposeCompounds is true. (ETROG-3263, ETROG-3264, ETROG-3265)

[en] Performance improvement: English and Spanish disambiguation with is now faster. Alternate disambiguation (alternateEnglishDisambiguation or alternateSpanishDisambiguation) must be set to false. (ETROG-3246, ETROG-3243)

[en] In Hebrew, prefixes in some acronym tokens are now listed correctly in the list of prefixes, instead of being duplicated in the lemma. (ETROG-3214)

[en] Example: “ומש"ס”

[en] Sentence breaks are now correct when there are two line breaks and fragmentBoundaryDetection is enabled. (ETROG-3241)

[en] Example: "a very very very very long line\nshort\n\n"

[en] In Hebrew, lemmas starting or ending with spaces now have the spaces removed. (ETROG-3248)

[en] Example: "אאורקה"

[en] Analysis of unknown Hebrew words with guessed prefixes no longer have duplicate prefixes in their prefix list. (ETROG-3253)

[en] Example: "בפיירפוקס"

[en] In Chinese and Japanese, the system no longer crashes when both fragmentBoundaryDetection and alternativeTokenization are enabled. (ETROG-3260)

[en] In Japanese, adjacent tokens are no longer erroneously joined when alternativeTokenization is enabled. (ETROG-3261)

[en] When universalPosTags are enabled the UPT-16 POS tags are now marked as having the tag set UPT16_V1 instead of the default tag set of the language. (ETROG-3273)

[en] Example: French

[en] We've fixed the tokenize-analyze example in the samples directory. It now correctly produces results for Hebrew analysis. (ETROG-3252)

[en] Layout regions added: Layout regions, describing each section of input text as STRUCTURED or UNSTRUCTURED, are now identified by the annotator. In order to detect layout regions, fragment boundary detection must be enabled. (ETROG-3172)

[en] New short line parameter: The option maxTokensForShortLine has been added to configure how many tokens can be in a line for it to be considered short for fragment boundary detection. The default value is 6. (ETROG-3179)

[en] Greek time abbreviations: The time abbreviations "π.μ." and "μ.μ." are now identified and annotated in Greek. The option fstTokenize must be set to true. (ETROG-3226)

[en] Greek coverage expanded: POS tags and lemmas are now recognized for some Greek words previously not identified. (ETROG-3225)

[en] Hebrew user-defined dictionaries added: Static and dynamic user-defined Hebrew analysis dictionaries are now supported. (ETROG-3230)

[en] Deprecated method: HebrewAnalysis#characteristicString is now deprecated. (ETROG-3209)

[en] Order of user-defined dictionaries: The order in which user-defined dictionaries are consulted has been standardized. Refer to the RBL-JE Application Developer's Guide for details. (ETROG-3148)

[en] Whitespace-delimited fragment boundaries are no longer skipped when they fall within tokens. This only occurred when fstTokenize was enabled and in some languages. (ETROG-3159)

[en] Example: "1\n234" (embedded newline within the number string)

[en] This example assumes fstTokenize is enabled and the language is French.

[en] Fragment detection now counts tokens correctly to determine short lines. This mostly impacts languages without spaces: Chinese, Japanese, and Thai. (ETROG-3177)

[en] Tokens with digits are now eligible for the Greek guesser. (ETROG-3231)

[en] In Hebrew, tokens with an unknown part of speech are no longer assigned the part of speech of one of their prefixes. This only occured when the guessHebrewPrefixes option is set to true.(ETROG-3221)

[en] Example: "ומפיפרנו"

[en] Russian perfective verbs are now lemmatized correctly. Previously some were lemmatized to their imperfective counterparts' lemmas or other incorrect lemmas. (ETROG-3112)

[en] Example: "разложу" where "разложу" is perfective and its lemma is "разложить". Its imperfective counterpart’s lemma is "раскладывать"

[en] German lemmas that consist of a separable prefix and a noun are now correctly capitalized. (ETROG-3235)

[en] Example: Input "Mitbehandlung"; "mit" is a separable prefix

[en] In Hebrew, terminal combining characters are no longer getting split into their own tokens. (ETROG-3224)

[en] Example: "1" (keycap)

[en] Hebrew tokens that have prefixes but not stems now get appropriate parts of speech. Previously, they got the POS tag "unknown". (ETROG-3207)

[en] Example: “ה” from the string “ה70”

[en] Lucene/Solr up through version 8.5.1 is now supported. (ETROG-3208)

[en] When guessHebrewPrefixes is true, unrecognized Hebrew tokens will now get analyses with and without potential prefixes. Previously, they would only get analyses with potential prefixes. (ETROG-3188)

[en] Example: Token: "ומפיפרנו"

[en] Minimally-qualified emoji are no longer split apart. (ETROG-3185)

[en] Example: The emoji for "man tipping hand" (<U+1F481, U+200D, U+2642>: )

[en] Capitalized nouns are no longer being detected as verbs. (ETROG-3186)

[en] Example: The noun "Service" from the phrase "Price and Quality of Service"

[en] When creating multiple analyzers for Chinese, Japanese, or Thai with alternateTokenization set to false (the default), the analyzers will now share the same model data. This will improve memory usage when creating multiple analyzers. (ETROG-3200)

[en] Note: While memory usage has been improved, the process is still memory intensive. If RBL throws an OutOfMemoryError, increase the heap space.

[en] We removed some incorrect entries from the Hebrew lexicon that were added in 7.34.0.c62.2. (ETROG-3182, ETROG-3183)

[en] Lucene/Solr: RBL-JE now supports Lucene/Solr up through version 8.4.1. (ETROG-3156)

[en] Unicode 13.0 emojis: Unicode 13.0 emoji sequences are now tokenized. (ETROG-3164)

[en] Additional emoji support: Emoji hair components are now lemmatized. (ETROG-3167)

[en] German professions: Additional German professions have been added to the German lexicon. (ETROG-3163)

[en] Spanish performance improvements: Spanish disambiguation is now faster when alternativeSpanishDisambiguation is false. (ETROG-3169)

[en] Hebrew lemmatization: We increased proper noun coverage in the Hebrew lexicon. (ETROG-3161, ETROG-3162)

[en] Low surrogates are no longer stripped from the ends of tokens in Hebrew. (ETROG-3165)

[en] Number tokens with embedded spaces are no longer split into multiple tokens when preceded or followed by a symbol when fstTokenize is true. (ETROG-3158)

[en] The delimiters for the fragment boundary detector are now configurable. (ETROG-3116)

[en] The fragment boundary detector now marks a boundary after any spaces following the fragment boundary delimiter. (ETROG-3116)

[en] An underscore (U+005F) is no longer treated as a token separator in German when fstTokenize is enabled. (ETROG-3144)

[en] We fixed a bug where tokens from multi-script Russian text sometimes had incorrect offsets if fstTokenize was enabled. (ETROG-3142)

[en] We fixed a bug where multi-script Russian text would have a sentence break each time the script changed. (ETROG-3145)

[en] We fixed a bug where there were unexpected sentence breaks after some short lines not ending in whitespace. (ETROG-3146)

[en] We fixed a bug where sentence breaks were missing when the sentence break did not align with a token boundary. (ETROG-3140)

[en] Added support for Lucene/Solr up through version 8.3.0. (ETROG-3128)

[en] Added support for tokenizing and lemmatizing Latvian. (ETROG-2798)

[en] Latin-script regions within Russian documents are now tokenized and analyzed as English. (ETROG-3126)

[en] TokenizerOption.licenseString, AnalyzerOption.licenseString, and BaseLinguisticsOption.licenseString may now be passed into a create method. Previously, these options had to be set on the factory itself. (ETROG-3134)

[en] We fixed a bug where guessed German compounds were sometimes lemmatized as verbs but tagged as nouns. (ETROG-3094)

[en] We fixed a bug where the fragment boundary detector would mark a sentence break after every Windows newline. (ETROG-3133)

[en] The Hebrew files dinflections.bin, dprefixes.data, and gimatria.data have been moved from the root/models directory to root/dicts/heb. (ETROG-3088)

[en] Specifying the universalPosTags option now adds the deliverExtendedTags option as well. (ETROG-2185)

[en] Dynamic user dictionaries can now be created and populated at runtime. See the section User-Defined Dictionaries in the Application Developer's Guide for details. (ETROG-3086, ETROG-3100, ETROG-3109, ETROG-3110, ETROG-3111)

[en] Fragment boundary detection is now enabled by default. Previously it was disabled by default. (ETROG-3108)

[en] TokenizerOption.alternativeTokenizationOptions has been deprecated in favor of a separate options for each YAML key. See the Javadoc for details. (ETROG-3109)

[en] The UPT-16 files upt-16-pes.yaml and upt-16-prs.yaml have been removed from the distribution package, as they were unused. (ETROG-3122)

[en] The -order option in rbl-build-csc-dictionary has been removed. All dictionaries are now built as LE, as LE dictionaries still work on BE machines. (ETROG-3120)

[en] We've added imperative forms for 2000 verbs to the Arabic lexicon. (ETROG-3090)

[en] Fragment boundary detection is now enabled for Hebrew. (ETROG-1442)

[en] When lemmatizing numbers in Russian, numbers containing spaces will now be lemmatized without the space. For example, "1 234" will now be lemmatized as "1234" instead of "1 234". (ETROG-3101)

[en] We fixed a bug introduced in 7.30.1.c61.0 which raised an ArrayIndexOutOfBoundsException when processing Japanese with alternativeTokenization and favorUserDictionary set to true. (ETROG-3118)

[en] We fixed a bug where a middle dot would be ignored if it preceded white space when using alternativeTokenization in Japanese. (ETROG-3113)

[en] We fixed a bug where an AssertionError might be thrown when analyzing Hungarian with Java assertions enabled.

[en] Russian words hyphenated with a number are now tagged with the part of speech of the word without the number.

[en] Correct token offsets are now returned from a Japanese annotator where a non-katakana character precedes a user-defined katakana token and alternativeTokenization and favorUserDictionary are enabled.

[en] We fixed a bug where constructors of factory classes in the Lucene/Solr plugin would throw an UnsupportedOperationException if passed a Map that did not support the remove method.

バージョン8.2.0までLucene / Solrのサポートを追加しました。

ビッグエンディアンとリトルエンディアンの両方の装置で使用される辞書とモデルのファイル名にLE/BEの定義が含まれなくなりました。

alternativeTokenizationがtrueに設定されている場合、中国のトークナイザーは、コンテキストを確認せずに入力文最後の品詞をNTと示す場合がありましたが修正しました。

alternativeTokenizationを有効にした中国語と日本語の分析速度が、数千文字以上の文に対して高速になりました。

セグメンテーションユーザー辞書は、中国語、日本語、タイ語だけでなく、すべての言語で使用できます。

オプションのcompoundComponentSurfaceFormsが追加され、複合語のコンポーネントの表層系が返されます。 デフォルトでは、RBL-JEはレンマのみを返します。

バージョン8.1.1までのLucene / Solrを対応しました。

「-cku」、「-ska」、と「-sku」で終わるポーランド語の単語のレンマは「-cki」または「-ski」で終わる形式になりました。

日本のPOSタグNEはUPT-16に正しく変換されるようになりました。

フランス語のPOSタグCONJQUEはUPT-16 CONJに変換されていましたが、より適切なSCONJに変換されます。

alternativeTokenizationが無効な場合、中国語の句読点をGUESSとしてタグ付けしていましたがPUNCTまたはEOSとしてタグ付けされるように変更しました。

タイ語でAlternativeTokenizationをtrueに設定すると、RLPパッケージ (C++) 同等の分ち書きプロセッサーを使用できます。(ETROG-2995)

ヘブライ語の品詞出力を改善しました。句読点の追加、そしてこれまで品詞が空白の単語はUNKNOWNと定義されます。(ETROG-3010)

U+05F3ヘブライ語句読点（geresh）とU+05F4ヘブライ語句読点（gershayim）の正規化に合わせるために、現在U+2019右シングルクォーテーションマークとU+201D右ダブルクォーテーションマークがヘブライ語のU+0027アポストロフィとU+0022クォーテーションマークに正規化されています。

英語の用語集を更新しました。

バージョン8.0.0でLucene/Solrのサポートを追加しました。

ドイツ語の用語集を更新しました。

スウェーデン語の用語集を更新しました。

認識されないトークンの場合、アラビア語解析によってハムザ（声門閉鎖音）の記号で表記される主要なアレフがプレーンのアレフに置換されます。

「מה」などの語基が含まれない複数の接頭辞で構成されるヘブライ語トークンの表面形式は、単に1番目の接頭辞ではなく、全体的なトークンテキストとなりました。

「Аполлона-11」など、末尾に数字がくるハイフンでつなげたロシア語の単語は、DIGとしてタグ付けされなくなりました。7.27.2.c60.0以前と同じ品詞でタグ付けされます。

URLが有効時にURLの後に続く右小括弧、右大括弧、右中括弧は、URLに統合されなくなりました。

曖昧性解消モジュールによって、ATMENTION、EMAIL、HASHTAG、URLのPOSタグ付き解析が、他の解析よりも優先して選択される可能性が高くなりました。

ヘブライ語トークナイザーが、ヘブライ語で使用される文字のすぐ後のヘブライ語で使用されない文字に遭遇すると、新しいトークンを開始します。以前は、その文字と、次のトークンセパレーター（空白など）までのその文字以降の文字が削除されていました。

RBL-JEが、BOMで始まるICUトークン化ルールファイルを正常に読み込めるようになりました。

語基が含まれない複数の接頭辞で構成されるヘブライ語トークンは、単一接頭辞トークンに合わせて、品詞「不明」でタグ付けされるようになりました。

英語トークン「than」は、COTHANとしてのみタグ付けされます。品詞候補のCOORDは、このトークンから削除されました。

ヘブライ語に対し、パーセプトロンベースの曖昧性解消器が使用できるように変更しました。 デフォルトでオプションDisambiguatorTypeがDisambiguatorType.PERCEPTRONに設定されています。以前のプロセッサーよりLemmaと品詞の分析精度が向上しています。 以前の設定に戻すには、DisambiguatorTypeをDisambiguatorType.DICTIONARYにして下さい。(ETROG-2985, ETROG-3006)

Lucene / Solrのサポートをバージョン7.6.0まで追加しました。(ETROG-2996)

Java 11 に対応しました。(ETROG-2999)

代替トークン化が有効時に、一部の空白文字が中国語トークンの一部であった可能性があります。

文字の長さが何千にもなるトークンは、トークナイザーを遅延させます。

複単語表現に見られるポーランド語トークンは、完全表記の範囲まで見出し語化されなくなりました。たとえば、「dzień」は「dzień_dobry」として見出し語化されません。

1つ以上のハイフンでつなげたロシア語の複合語の非最終文節は、見出し語化されていませんでした。偶然にも形容詞の短縮形に見えた接合辞「е」または「о」のある、ハイフンでつなげたロシア語の複合語の非最終文節は、短縮形のごとく見出し語化されていました。