Manage

Note

You must be registered as a manager.

Manager options can be accessed from the Manage option on the project navigation bar.

Management

The following management tasks and information are accessed by selecting Manage from the navigation bar.

Users: This section allows you to add and remove annotators and adjudicators from the project. You can also see how many samples have been assigned to each annotator, as well as how many they have completed.
Documents: A list of all uploaded documents along with assigned annotators and annotation statistics. For each document you can copy the Doc ID, view the original document, rename the document, reassign annotators and adjudicators, or delete the document from the project. You can also edit annotator and adjudicator assignment on a project level by selecting Annotator assignment or Adjudicator assignment at the top of the Documents panel.
Model Training Status: Displays project statuses and allows you to export the model being trained.
- Status of RTS (REX Training Server), ETS (Events Training Server), and Rosette Server:
  - A green checkmark indicates the server is up and connected.
  - A red exclamation point indicates the server is down or not connected.
  - A gray exclamation point with a slash through it indicates the server is not installed.
- License Expiration: The expiration date of your license.
- Model: The status of the model being trained.
- Word classes: (NER models only) Shows whether word classes have been generated and are available. The model can be exported when the word classes are not available.
- Project: The reconciliation status. When reconciled, the confidence scores for each sample, along with the precision, recall, and F1 measure are current. When unreconciled, the scores may be inconsistent with the latest trained model. To reconcile the model immediately, select Reconcile from the project menu. This task is run every night.
- Export model: This button downloads the model trained from the annotations in the project. This file can then be loaded into Rosette Server in your production environment to deploy the model. The option is disabled while the model is actively training and being written to disk. It will become enabled once the model is ready for export, which is approximately 10 minutes after the last annotations.
  - An NER model may be exported before word classes are available.
  - Tentative extractors should be resolved before exporting an event model.
    
    Note
    
    Samples with tentative transactors are not used for training. If you try to export a model that contains tentative extractors, a message will appear warning you that the model contains tentative extractors.
Exported Versions: A list of projects exported. You can download or delete the export files from here. This section is only visible when there is at least one exported project.
Clear Annotations: Remove all annotations from one or more annotators for a selected entity or event type. You can use this feature to remove the work of annotators who have very low agreement with other annotators from the project. For more information on annotator agreement and its importance to model training, see Inter-Annotator Agreement.

Note

Clearing annotations via the Manage page changes the status of all affected samples to unannotated.
Guidelines File: Add the URL of your written annotation or other project guidelines. The guidelines can be in any format, such as google doc, pdf, or html page. The guidelines should be available to all users of the Studio. The url should include the full web address, including http/https.

Add Annotators or Adjudicators to Project

Select Manage from the project navigation bar.
Select Add/edit roles and permissions.
Select the checkbox under Annotate next to the desired user if you want them to be an annotator.
Select the checkbox under Adjudicate next to the desired user if you want them to be an adjudicator.
Select Save.

Remove Annotators or Adjudicators from Project

Select Manage from the project navigation bar.
Select Add/edit roles and permissions.
De-select the checkbox under Annotate next to the desired user if you want to remove them as an annotator.
De-select the checkbox under Adjudicate next to the desired user if you want to remove them as an adjudicator.
Select Save.

Note

If you remove an annotator who has already made annotations, their annotations will disappear and Adaptation Studio will display the message "Project includes orphaned annotations by (user)" beneath the list of annotators. If you re-add this user, their annotations will return to the project.

Project Level Annotator Assignment

For projects containing a large volume of annotators and documents, it is easiest to edit annotator assignments on a project level. For example, if you want to add a new annotator to a project after many documents have already been added, you can use this feature to quickly assign all documents to that annotator.

Select Manage on the project navigation bar.
Select Annotator assignment at the top of the Documents panel.
Select a checkbox to assign the corresponding document to that annotator.

Note

Select Assign all or Unassign all in the rightmost column of a document's row to cycle between selecting and deselecting all annotators for that document. Select an annotator's name to cycle between selecting and deselecting all documents for that annotator, including documents on other pages of the table.
Deselect a checkbox to unassign the corresponding document from that annotator.
Select Save to make your changes effective.

Note

If you want to discard your changes, you can select Back to return to the Manage screen without making your changes effective.

Project Level Adjudicator Assignment

If you want to add a new adjudicator to a project after many documents have already been added, you can use this feature to quickly assign documents to that adjudicator.

Select Manage on the Project Navigation Bar.
Select Adjudicator assignment at the top of the Documents panel.
Select a checkbox to assign the corresponding document to that adjudicator.

Note

Select Assign all or Unassign all in the rightmost column of a document's row to cycle between selecting and deselecting all adjudicators for that document. Select on an adjudicator's name to cycle between selecting and deselecting all documents for that adjudicator, including documents on other pages of the table.
Deselect a checkbox to unassign the corresponding document from that annotator.
Select Save to make your changes effective.

Note

If you want to discard your changes, you can select Back to return to the Manage screen without making your changes effective.

Add Annotators or Adjudicators to Document

Note

A user must be added to the project as an annotator before they can be added to a document as an annotator. A user must be added to the project as an adjudicator before they can be added to a document as an adjudicator.

Select Manage from the project navigation bar.
Select the pencil icon in the Action column for the desired document in the Documents panel.
Select the checkbox under Annotate next to the desired user if you want them to be an annotator.
Select the checkbox under Adjudicator next to the desired user if you want them to be an adjudicator.
Select Save.

Remove Annotators or Adjudicators from Document

Select Manage from the project navigation bar.
Select the pencil icon in the Action column for the desired document in the Documents panel.
De-select the checkbox under Annotate next to the desired user if you want to remove them as an annotator.
De-select the checkbox under Adjudicate next to the desired user if you want to remove them as an adjudicator.
Select Save.

Note

If you remove an annotator who has already made annotations, their annotations will disappear. If you re-add this user, their annotations will return to the project.

Add New Users to Existing Project

These steps show how to add completely new users to an existing project that has already been annotated.

Add the new users to the system. You can think of this as creating accounts for new users on your instance of Adaptation Studio.

Select User Management from the global navigation bar.
Select Add user to create an id for each user. Assign a role to each user.

Add the new users to the project. This gives your new users annotator- or adjudicator-level access to the project you want them to work on.

Select All Projects from the global navigation bar to display all projects you have access to.
Select the project you want to add the new user(s) to.
Select Manage from the project navigation bar.
Select Add/edit roles and permissions.
- Select the checkbox under Annotate next to the desired user if you want them to be an annotator.
- Select the checkbox under Adjudicate next to the desired user if you want them to be an adjudicator.
Select Done.

Assign documents to the new users. This gives your new annotators and adjudicators the ability to annotate and adjudicate in the project.

Scroll down and select Annotator assignment at the top of the Documents panel. Adaptation Studio displays the annotator assignment table. Each new user is represented by a column of empty checkboxes with their name at the top. The checkboxes are empty because they do not yet have any documents assigned.
For each new annotator, select the checkbox in the row of each document you want to assign them. You can select the user's name at the top of their column to select and de-select all documents for that user.
Select Save.
Select Adjudicator assignment at the top of the Documents panel.
For each new adjudicator, select the checkbox in the row of each document you want to assign them. You can select the user's name at the top of their column to cycle between selecting and de-selecting all documents for that user.
Select Save.

Export Models

Note

If using Chrome, you must disable popup blocking to allow multiple files to download.

Once a model has been trained, you can move it from the training server to your production instance of Rosette Server. The trained models are used by Rosette Server endpoints.

The models trained to perform entity extraction are consumed by REX and the entities endpoint.
The models trained to perform event extraction are consumed by the events endpoint.

To export a model:

Open the project that trained the model you are interested in.
Select Manage from the project navigation bar.
From the Model Training Status block, select Export Model.

Tip

If Export Model is not enabled, the model is not ready to be exported.

The trained model will download to your machine.

Note

Samples with tentative transactors are not used for training. If you try to export a model that contains tentative extractors, a message will appear warning you that the model contains tentative extractors.

Two files are downloaded when the model is exported. The model name is part of the file name for both files.

The model file. This is the file that gets uploaded to the production server.
An information file with information about the model. This is a json text file.

Example 1. Sample NER Export

Files:

rts-model.6178335ba1695d93f64553fe-20211122_11_39_09-LE.bin

rts-model.6178335ba1695d93f64553fe-20211122_11_39_09-LE.bin.export-info

Info File:

{
  "export_time": "2021-11-15 11:37:49",
  "language": "eng",
  "labels": [
    {
      "type": "PERSON",
      "label": "Person"
    },
    {
      "type": "LOCATION",
      "label": "Location"
    },
    {
      "type": "ORGANIZATION",
      "label": "Organization"
    },
    {
      "type": "TITLE",
      "label": "Title"
    },
    {
      "type": "PRODUCT",
      "label": "Product"
    }
  ],
  "training_count": 795,
  "eval_count": 75,
  "scores": {
    "precision": 1,
    "recall": 1,
    "f1": 1
  }
}

Complete NER scores will be available in a future release.

Example 2. Sample Event Export

Files:

ets-model.616d2e0c5718dde327bfc11f-20211122_11_38_46-LE.ets-model

ets-model.616d2e0c5718dde327bfc11f-20211122_11_38_46-LE.ets-model-info

Info File:

{
  "export_time": "2021-11-18 08:17:53",
  "language": "eng",
  "labels": [
    {
      "type": "key",
      "label": "flight_booking"
    },
    {
      "type": "origin",
      "label": "origin"
    },
    {
      "type": "when",
      "label": "when"
    },
    {
      "type": "destination",
      "label": "destination"
    }
  ],
  "training_count": 59,
  "eval_count": 56,
  "scores": {
    "precision": 0.4,
    "recall": 0.47058823529411764,
    "f1": 0.4324324324324324
  }
}

Project Menu Options

These tasks are available from the project menu in the upper right-hand corner of the project dashboard.

Show Guidelines: Displays the uploaded project guidelines.
Rename: Renames the project.
Clone: Creates a copy of the project. This is equivalent to exporting and then importing the project.
Configure: Modify the project configuration options.
Reconcile: Recalculates the confidence scores for all the samples in addition to the precision, recall, and F1 measure. As this can be time intensive, the scores are recalculated every night. The reconcile task allows you to update scores at any time.
Export Project: Creates a backup file for the project. Once exported, the project will be listed in the Exported Versions section of the Manage page. From here, you can download or delete the file.
Extract ADMs: Downloads a zip file of .json files containing the annotations in ADM format with one file for each document.

These .adm files contain much useful information about the annotation work that was done, including each of the annotations, which annotator performed each, and timestamps.

These additional options are only available when in the all projects view, not when a project is open:

Delete Docs: Deletes all documents from a project.
Delete: Deletes the project.

Export Model, Export Project, and Extract ADMs

The Studio has multiple ways of downloading models and annotations.

Export Model: Downloads trained models from the Studio to the local machine. Two files are downloaded:
- The .bin file is the trained model which you then upload to a production server to use the model for extraction.
- The export-info file is a .json text file containing information about the model.
Select the Export model button in the Manage page.
Export Project: Exports a project to the server as a zip file. This file can then be used to reload the project to the same or a new instance of the studio. The directories in the zip file are:
- project: ADM files for each sample in the project.
- model: The trained model.
- documents: The sample documents.
Select Export Project from the project menu. The .zip file can be downloaded from the Exported Versions table in the Manage page.
Extract ADMs: Exports to the server a zip file containing all annotated and adjudicated samples in json ADM format. Each annotator's work is in a separate directory.

Select Extract ADMs from the project menu. The .zip file can be downloaded from the Exported Versions table in the Manage page.

Articles in this section

Note

Management

Note

Note

Add Annotators or Adjudicators to Project

Remove Annotators or Adjudicators from Project

Note

Project Level Annotator Assignment

Note

Note

Project Level Adjudicator Assignment

Note

Note

Add Annotators or Adjudicators to Document

Note

Remove Annotators or Adjudicators from Document

Note

Add New Users to Existing Project

Export Models

Note

Tip

Note

Project Menu Options

Export Model, Export Project, and Extract ADMs

Comments

Articles in this section

Note

Management

Note

Note

Add Annotators or Adjudicators to Project

Remove Annotators or Adjudicators from Project

Note

Project Level Annotator Assignment

Note

Note

Project Level Adjudicator Assignment

Note

Note

Add Annotators or Adjudicators to Document

Note

Remove Annotators or Adjudicators from Document

Note

Add New Users to Existing Project

Export Models

Note

Tip

Note

Project Menu Options

Export Model, Export Project, and Extract ADMs

Related articles