リリースノート

[en] We fixed a linking problem introduced in 7.55.0.c68.0, in which the NIL_BIAS parameter value was incorrect. The parameter is now correct for all languages. (TEJ-1899)

[en] Wikidata refreshed: We've updated the knowledge base data for the provided linking knowledge base. The QID assigned to some extracted entities may differ from previous versions. (RWIki-119, ELK-274, ELK-276)

[en] New currency regex: We've introduced a new option, regexCurrencySplit, that, when set to true, will attempt to split entities extracted with the regex engine of type IDENTIFIER:MONEY into two new entities: IDENTIFIER:CURRENCY_AMT and IDENTIFIER:CURRENCY_TYPE. These two new types represent the amount of the currency (50,000) and the currency type ($), respectively. By default, regexCurrencySplit is set to false. (TEJ-1792)

[en] Tagalog support: We've added case-insensitive NER support for Tagalog. Previously we released a case-sensitive model and we've now added the case-insensitive model as well. (TEJ-1858)

[en] Parameter removed: We've removed the deprecated genre extraction option. This option was used to turn the linker on which has been, and will still be, available by the linkEntities option. The genre option is no longer available in the REX SDK, in the Rosette Server REX configuration, as well as the Rosette API bindings (TEJ-1855).

[en] This beta release contains a new English statistical extraction model which was trained on financial data. The model has improved accuracy over the default English models for financial domain documents. (TEJ-1837)

[en] Tagalog (tgl) support: We've added Tagalog to our list of languages. The following processors are supported: gazetteer, regex, statistical NER, linking. (TEJ-1812, TEJ-1822, TEJ-1785, TEJ-1786)

[en] New linking option: We've added a new option for entity linking. When linkMentionMode is set to entities the linker will attempt to link the entities extracted by other processors (regex, gazetters, and the statistical processor) instead of using its own processor to extract entity candidates. Depending on your data, this may provide higher accuracy and speed. (TEJ-1806)

[en] REXCmd parameter change: The linkEntities parameter can now act as a toggle instead of taking a true/false value, matching how other REXCmd boolean parameters are handled. (TEJ-1806)

[en] Parameter deprecated: The parameter genre is deprecated and will be removed in the next release.

[en] REX no longer produces an exception when token normalization produces an empty token string. (TEJ-1803)

[en] When looking for candidate mentions in text, if there is an overlap between these mentions the linker now resolves the longest spanning mention before disambiguation. (ELK-277)

[en] Configure knowledge base linking priority: With multiple knowledge bases it is possible to set the order in which to try linking against each knowledge base. Set the priority in the redactor configuration file (ne_types.xml) (TEJ-1726, TEJ-1754)

[en] Example: The following XML element will set the custom-kb priority higher than the default knowledge base (kb-linker) when linking a PRODUCT entity type:

<ne_type>
  <name>PRODUCT</name>
  <weight name="kb-linker" value="100" />
  <weight name="kb-linker:custom-kb" value="1" />
</ne_type>

[en] relatedEntities renamed to contextWords: When creating a custom knowledge base, the feature contextWords, which was previously called relatedEntities, is required. Context words are language-specific words that are strongly related to the entity. The term relatedEntities has been deprecated. (TEJ-1756)

[en] Java 17 support added: Java 8 and 9 support has been removed. (TEJ-1728, TEJ-1763)

[en] Solr 9 support added: REX now supports Lucene and Solr 9. (TEJ-1731)

[en] Solr 6 support deprecated: REX no longer supports Solr 6 or earlier. (TEJ-1731)

[en] Bug fix: An error is no longer generated when there are null prefixes in Arabic morphological analyses. (TEJ-1765)

[en] Bug fix: We fixed a bug to enable using noisy_context_vector feature for disambiguation. (ELK-265, ELK-268, ELS-272, TEJ-1776)

[en] Updated Log4j to version 2.17.1. (TEJ-1724)

[en] New RBL version: Entity extraction now consumes the latest version of Rosette Base Linguistics (RBL) 7.42.2.c65.0. (TEJ-1681, TEJ-1693)

[en] Hungarian dates are now extracted correctly. Previously, dates with embedded periods followed by a space were not being extracted. (TEJ-1681)

[en] rexcmd info no longer lists TEMPORAL types by default for SWEDISH. (TEJ-1687)

[en] The supported entity types info of the DNN processor now spells PERSON correctly. (TEJ-1670)

[en] Wikidata refreshed: The internal database for Wikidata linking has been refreshed and re-indexed. QIDs for some entities may change from previous versions. (TEJ-1657, TEJ-1658)

[en] New RBL version: Entity extraction now consumes the latest version of Rosette Base Linguistics (RBL) 7.41.1.c65.0. (TEJ-1667)

[en] A single line followed by an empty line is no longer always considered a fragment. (ETROG-3431)

[en] The Field Training Kit (FTK) no longer returns erroneous error messages from generating wordclasses. (TEJ-1636)

[en] The RBL models directory is now correctly specified in the FTK. (TEJ-1655)

[en] The REX Training Server (RTS) no longer fails when the request contains the language code msa. msa is now mapped to zsm, the language code supported by REX for Malay. (TEJ-1669)

[en] We fixed a bug where Invalid whitespace handling by the DNN processor would cause a runtime exception. (TEJ-1614)

[en] Language-specific joiner rules: Custom joiner rules can now be language-specific or apply to all languages. (TEJ-178)

[en] New default processing for structured text regions (lists, tables): Because structured text is often just words or phrases, and thus missing the syntactic context that REX was trained on, some REX users would pre-process input text to remove structured regions, on which REX performed poorly. Users no longer have to pre-process the input as now the statistical/DNN model is turned off by default for structured regions. This mode increases precision but may result in reduced recall in these regions. Note, the other REX processors (pattern match, exact match, entity linking) which do not rely on context will continue to analyze the structured regions. To turn on the statistical/DNN model for structured regions, set the parameter structuredRegionProcessingType to nerModel. (TEJ-1502) (TEJ-1502)

[en] New name classifier model for structured regions (LABS): We've added a new model for processing structured regions. The name classifier classifies a text fragment as PERSON, LOCATION, ORGANIZATION, or NONE. The entire structured region is classified as a single label, an entity type or NONE. It is disabled by default. (TEJ-1613, TEJ-1621)

[en] Japanese organization gazetteers: The gazetteers for Japanese organizations has been updated to improve extraction of Japanese organizations. (TEJ-1612)

[en] New RBL version: Entity extraction now consumes the latest version of Rosette Base Linguistics (RBL) 7.39.0. (TEJ-1618)

[en] Rosette Training Server (RTS) results: When using REX with Rosette Adaptation Studio (RAS), the results returned by RTS are now preferred by default. (TEJ-1605)

[en] Entities are no longer extracted when they cross a sentence boundary. To enable entity linking across sentence boundaries, set disableApplySentenceBoundaries to true. (ELK-259)

[en] Entities are now checked to ensure they are normalized. (TEJ-1615)

[en] We fixed an offset alignment issue for carriage return normalization. (TEJ-1600)

[en] Updated the internal database for Wikidata linking. QIDs for some entities may change from previous versions, as Wikidata has been refreshed and re-indexed. (TEJ-1579, ELK-249, ELK-251, RWIKI-77)

[en] Updated RBL version (TEJ-1579)

[en] The sqlite-kb-connector sample now works correctly. Runtime issues with sqlite dependencies have been corrected. (ELK-245, ELK-257)

[en] Extraction no longer fails when a custom processor returns a NULL annotator; instead a warning is generated. (TEJ-1580)

[en] Mentions normalized by the custom processor are no longer ignored. (TEJ-1573)

[en] Windows-formatted carriage returns (/r, /r/n) are now handled correctly.

[en] Joiner runs before redactor: The joiner now runs before the redactor by default, providing more flexibility and control over the  joiner results. Set runJoinerPostRedactor to true to run the joiner after the redactor. (TEJ-1534)

[en] Improved phone number recognition: Regular expressions for phone number extraction have been improved and now extract more phone number patterns. (TEJ-1556)

[en] REXCmd input from stdin: REXCmd can now accept input from stdin by specifying the command line option -stdin.

[en] Example:

$ echo "Basis Technology is a company in Massachusetts" | REXCmd extract -stdin -langCode eng

[en] We fixed a bug where sometimes a null pointer exception was returned when the custom processor and the linker had overlapping results. (TEJ-1561)

[en] Custom processors can now only modify the entity and metadata sections of the ADM. Previously, any modification could be made which could override annotation data. (TEJ-1537)

[en] We've partially fixed a problem in Japanese ORG extraction where sometimes the model extracts multiple ORG entities or includes non-related adjacent tokens. (TEJ-1534)

[en] The Field Training Kit no longer generates invalid models for when creating custom knowledge bases. This occurred for all languages except eng, jpn, and zho. (ELK-252)

[en] Improved sample The sample files to build the SQLite connector described in the Custom Knowledge Base Connectors section now includes all files required to build with Maven. The configuration to run the connector with Rosette Enterprise is now provided as well. (TEJ-1508)

[en] Language-specific alias Custom knowledge bases compiled with the Field Training Kit (FTK) will now maintain the language of the alias. Aliases will only be extracted in documents of the language the alias is defined for. Aliases can be defined as for all languages or for a specific language. (ELK-241)

[en] Custom knowledge bases can be compiled without disambiguation. While adding a knowledge base without a disambiguation model will not provide the best results, it will function as an enhanced gazetteer that attaches an assigned ID to each gazetteer entry and supports multiple aliases per entry. To compile a custom knowledge base without compiling a disambiguation model pass -d as an argument to train-linker-model. (ELK-233)

[en] New method A method getBaseLinguisticsParameters has been added to retrieve the base linguistics parameters that were used in training the model. Use the retrieved parameters to configure an external instance of RBL to produce tokens consistent with the training tokenization. A new sample application, RBLParametersSample.java, is available in the samples directory. (TEJ-1501)

[en] Base linguistics added The FTK can now use input ADM files containing base linguistics annotations, such as tokens, sentence boundaries, and morphological analysis for languages such as Korean and Arabic. For REX to produce the optimal results, tokenize with the options provided by the getBaseLinguisticsParameters method when creating the ADM file from RBL. (APE-1793)

[en] Hebrew improvements REX has improved Hebrew normalization and added the ability of the disambiguator to identify prefixes removed from the entity's normalized form. Improvements are a result of enhancements in Hebrew base linguistics. (ETROG-3189)

[en] A new line character in a regex (\n) will now also match carriage returns (\r) and a combination of both (\r\n). (TEJ-1525)

[en] Confidence scores for entity linking now use the same scale, whether linking to Wikidata or a custom knowledge base. Previously, the confidence scores given for links to custom knowledge bases were much lower than those calculated for the Wikidata knowledge base. (ELK-240)

[en] Connector framework for custom Knowledge Bases added. See section 5.6 in the Application Developer's Guide. (TEJ-1476, TEJ-1477, TEJ-1485)

[en] Added Deep Neural Network model for Hebrew for improved accuracy. Replace statistical model with it by using the flag -useDeepNeuralNetworkProcessor. (TEJ-1503)

[en] Hebrew normalization improved: instead of using the lemma form, just the prefixes are being removed, except the definite article. (TEJ-1505)

[en] New statistical model for Hebrew trained on news and finance data. (TEJ-1497)

[en] Solr plugin now available as a Docker container. (TEJ-1492)

[en] Supplemental regex support for ISO-6709 geo-coordinates. (TEJ-1431, DATA-761)

[en] Support for setting prioritization for multiple custom Knowledge Bases. See section 5.2 in the Application Developer's Guide. (ELK-236)

[en] Redactor weighs can now be configured for specific subsources. See section 3.2.1 in the Application Developer's Guide. (TEJ-1480)

[en] Separate license key required for linker custom Knowledge Bases. Note: extractions against existing custom Knowledge Bases will fail unless licenses are updated. (TEJ-1483)

[en] Custom Knowledge Bases can be set in Rosette Enterprise profiles. Note: To support this feature, the flinx directory was moved into {rex-installation}/data. Any custom data inside must also be moved to the new location. (TEJ-1494)

[en] TEJ-1499 REXAnnotatorFactory failed to assign linking confidence thresholds.

[en] TEJ-1479 Fixed dynamic gazetteers for Malay.

[en] TEJ-1506 Deep Neural Network extractions failed in REXCmd.

[en] TEJ-1470 Fixed an error when extracting entities in Arabic text.

[en] New language: Entity extraction now supports Swedish. (TEJ-1395)

[en] Manual cache memory eviction functionality added to the SDK. See the Application Developer Guide for details. (TEJ-1450)

[en] ELK-169 Fixed an error when reading aliases binaries produced with the FTK.

[en] ELK-179 FTK no longer requires English or Chinese models for other languages.

[en] Re-enabled and improved decompounding for Japanese and Chinese. It was disabled in 7.43.1.

日本語と中国語の複合化を無効にしました。

Java 11に対応しました。

ウィキデータのリンク用に内部データベースを更新しました。 DBPedia Typeフィールドは、複数のサブタイプをサポートするようになりました。 一部のエンティティのQIDは、Wikidataが更新されてインデックスが再作成されたため、以前のバージョンとは異なる場合があります。

エンティティリンクは、一部のエンティティのQID（Wikidata ID）に加えて、PermID（Thomson ReutersナレッジベースのID）を返します。

DNNの実行中に発生する可能性があるNULLポインター例外を修正しました。

ユーザ辞書Gazetteerのマルチスレッドバグを修正しました。

Flinx曖昧性解消モデルに実行時に制御するオプションのパラメーターファイルが前回供給したパッケージ内から欠落して精度不良が生じていましたが、再追加することで改善しました。

エンティティリンクで日本語文字が簡体字中国語に誤って正規化されたケースを修正しました。これは以前のリリースでも一部修正されていた問題です。

中国語ファイルシステムの言語コードは均一に3文字に修正しました。

エンティティ抽出とエンティティリンクは、最新バージョンのRBLを使用するように改善されました。

インストールをより簡単にするためドキュメントおよび言語パッケージの解凍およびインストールするスクリプトを追加しました。

日本の日付抽出に新しい年号の令和を追加しました。