Named entity extraction projects

Before starting annotation, add new labels, if necessary.

It may take a few minutes for an imported project to appear in the Project list, especially if it is a large project. The imported project will only be visible in the Project list to superusers and users assigned to the project.

Comments allow you to pose questions or provide clarifications about the sample being edited to adjudicators and managers. Adjudicators and managers can view the comments for each sample via View Annotations. Comments do not affect model training in any way. To add a comment while annotating, follow these steps:

You can edit comments you have made by selecting the commented annotation and selecting Edit Comment. From here you can also delete the comment by selecting the trash can icon.

When the Enable link annotation option is selected, you can add a link ID to the entity, along with the label. The link IDs are used to train a linking knowledge base. Rosette entity extractor supports linking to a default Wikidata knowledge base. You can also train it to link to a custom knowledge base with Model Training Suite.

The annotations in Rosette Adaptation Studio are sent to the REX Training Server and provide the data for training a new model. Only adjudicated annotations are included in training. The system will auto-adjudicate samples.

The rules for auto adjudication are:

Select Filters on the View Annotations page to expand the Filters panel. To view the filter criteria, click on the down arrow to expand the section. The Filters panel allows you to select which samples to display in the sample list using the following criteria:

Select Update results to filter the sample list so that it only displays samples with all chosen criteria.

All samples in the project that match the applied filters are listed below the filters panel.

Each sample is displayed next to a table showing the tags selected by each annotator for each entity. If you are a project manager, this view is useful for identifying and inspecting specific points of high annotator disagreement. Adjudicators and managers can also see any comments added by annotators on the rightmost column of the table.

The following are displayed below each sample:

If there are two or more identical entities, each will be labeled with a corresponding id in the sample text and the Entity column. Mouse over any entity in the Entity column to highlight that entity in the sample text.

The following management tasks and information are accessed by selecting Manage from the navigation bar.

{
  "export_time": "2021-11-15 11:37:49",
  "language": "eng",
  "labels": [
    {
      "type": "PERSON",
      "label": "Person"
    },
    {
      "type": "LOCATION",
      "label": "Location"
    },
    {
      "type": "ORGANIZATION",
      "label": "Organization"
    },
    {
      "type": "TITLE",
      "label": "Title"
    },
    {
      "type": "PRODUCT",
      "label": "Product"
    }
  ],
  "training_count": 795,
  "eval_count": 75,
  "scores": {
    "precision": 1,
    "recall": 1,
    "f1": 1
  }
}

These tasks are available from the project menu in the upper right-hand corner of the project dashboard.

The Studio has multiple ways of downloading models and annotations.

Provides information to project managers pertaining to the quantity and quality of the data being created by annotators and adjudicators. When training a model, it is critical to have the correct volume and distribution of high-quality data. Only managers have access to reports.

Most NLP tools are evaluated based on their precision and recall. Accuracy is measured as a combination of precision, recall, and F1 measure.

The correct measure depends on your application. In applications where you can only handle a few responses, such as voice applications (e.g. Amazon's Alexa), high precision with low recall is ideal, because the system can only present the user with a few options in a reasonable time frame. However, other applications, such as redacting text to remove personally identifiable information, redacting too much (low precision) is much better than missing even one item that should have been redacted. In that case, high recall is preferred.

The values displayed on the project dashboard are calculated using the annotated validation data as the gold data. As the model is trained, it generates new predictions and the scores are recalculated. The predictions generated by the model for the validation samples are compared with the annotated values in the samples.

Named entity recognition models are trained to recognize and extract entity mentions. Each annotated sample may contain multiple entities.

The statistics displayed on the project dashboard are the macro averages of the precision, recall, and F1 measures for the labels.

Let's look at how precision, recall, and f-score are calculated. Let's assume a search system, where a number of items were retrieved.

TP: True positive. Number of documents retrieved that are correct.

FP: False positive. Number of documents retrieved that are incorrect.

FN: False negative. Number of documents that should have been retrieved, but weren't.

Retrieved: All documents retrieved = TP + FP

Relevant: All documents that should have been retrieved = TP + FN

Precision is the fraction of correct retrievals among all retrieved instances.

Recall is the fraction of relevant documents that are successfully retrieved.

F-score is the harmonic mean of precision and recall

The project dashboard displays precision, recall, and F1 measures for the labels. There are two different ways of calculating these scores across multiple labels:

Machine-learning models are completely dependent on the quality of the data used for training. Inconsistent annotation or lack of adherence to the project annotation guidelines will lead to less accurate models. Especially when starting a new project or on-boarding new human annotators, check for reliable annotations by having a subset of data annotated in parallel by multiple human annotators.

Krippendorff’s alpha is a statistical inter-rater reliability metric used to measure inter-annotator agreement. Krippendorff’s alpha scores range from -1.0 to 1.0 with 1.0 indicating perfect agreement between annotators. A score of 0.0 indicates agreement no better than random chance (as if your annotators picked their labels randomly out of a hat). A reliability score of 0.80 or greater is generally considered sufficient (though, the higher the better). Lower scores may indicate potential issues with your data, your annotation guidelines, or your annotators’ understanding of the task. A low level of inter-annotator agreement will ultimately lead to a less accurate model, so we recommend repeatedly measuring the reliability of your annotators until they achieve a satisfactory level of agreement. The cases where annotators disagree are usually good examples to include in your annotation guidelines. It can be useful to have a discussion about points of disagreement with your annotators as a group to reach a consensus.

Adaptation Studio calculates the Krippendorff's alpha score as you adjudicate samples. To view the current values, select Manage from the project toolbar. The values will be displayed for all annotators and as pairs of annotators.

The IAA history report displays the progress of the inter-annotator agreement over time.