The Studio is a web-based tool. Your site administrator can provide you with the web address of your installation.
The main screen lists the projects you have access to. Superusers will see all projects created on the server.
If you are a manager or superuser, you can create new projects. Managers only see the projects they have created.
Click on a project to access the project dashboard. The dashboard displays the current state of the project and provides access to project-related tasks. Your user type will determine the tasks you can perform.
Common tasks are accessed from the navigation menu below the project status bar. Less-frequent tasks are accessed from the project (hamburger) menu in the upper right-hand corner of the project dashboard. Some tasks, such as Delete, are only available in the All Projects view.
When you log into Rosette Adaptation Studio, you will be one of the following types of users:
Superuser: creates users in addition to all other tasks
Manager: creates and manages new projects
Adjudicator: reviews annotations by multiple users and settles conflicts when samples are annotated differently
Annotator: annotates samples from documents in one or more projects
All user types can be annotators.
To add and manage users, select Admin Settings (superuser only).
Active learning is a form of supervised machine learning in which the learning algorithm guides the order of samples presented to the annotator. This can lead to significant reductions in annotation time and expense.
Active Learning accelerates the annotation process in two ways:
It presents the most uncertain sample for annotation, ensuring that the next sample you annotate will provide the most useful annotations.
The engine provides its best guess or suggestion for an annotation, where possible.
When you upload a document, it is divided into samples for annotation. A document is made up of multiple samples, where each sample is a sentence or paragraph in the document. The active learning module selects the "best" document for annotation. The best document is the one which has the lowest certainty for predicting annotations.
Active learning ensures that the samples containing unique or uncommon terms are annotated early in the process. Each time a sample is completed, the engine looks for the sample with the lowest confidence, that is the sample that is the most unfamiliar to the system. By selecting the unfamiliar samples, you prevent skewing of results towards more common and represented terms.
Tip
Annotators must be careful to review suggested annotations, especially early in the annotation process when they are more error-prone. You may want to suppress these suggestions to avoid biasing the human annotators. Set the configuration option Hide Eval Predictions to suppress suggested annotations for samples in the evaluation set. Suggestions will still be displayed for the training set.
Note
You must be registered as a manager for the project.
Annotation is a process of assigning labels to parts of documents. The set of labels, as well as the words or parts of words assigned to the label, depends on the type of annotation task.
When you create a project, a set of pre-defined labels may automatically be created for the project.
Named Entity Recognition (NER) labels are entity types. The predefined entity types are Location, Nationality, Organization, Person, Product, and Title.
Parts of Speech (POS) labels are a POS tag for a token or word in the selected language. Different languages will have different POS tags. The POS tags are predefined, but you can add or change them.
Event type labels are defined by the user.
To define a label, you supply a code, caption, and color:
Code: This is a brief name for the label. By convention, it is usually uppercase and about three letters. This will be used internally and in places where the UI shows a label in a brief form.
Caption: A longer user-friendly name for this label, usually one or two words long. It should be descriptive, but reasonably short.
Color: The color used for annotations of this label. You can accept the default color or assign a color.
You can add a label or edit existing labels after the project has been created.
To add a new label, select Add Label from the project menu, fill in the form, and Add.
To edit an existing label, click on the label on Project Dashboard. Change any of the fields and Save.
To delete a label, click on the label on the Project Dashboard and Delete. You can only delete a label if there are no annotations for it. Once the label has been assigned during annotations, it cannot be deleted from that project.
Warning
Once you've annotated samples, the Studio will not prompt you to re-annotate them when a new label is added.
Quick view of project status
The project dashboard displays several key metrics to give a high-level view of the status of the annotation effort. The metrics displayed are determined by the user's assigned tasks. Managers and superusers see all fields, annotators and adjudicators a subset.
Completed: (manager and superuser) This field displays how many samples have been annotated out of the total samples requiring annotation. It is counting sample/annotator pairs, so if five samples have each been assigned to two annotators, the total work is ten.
Adjudicated: (manager and superuser) This field displays how many samples have been adjudicated out of the samples requiring manual adjudication.
-
Samples:
annotators: The total number of samples assigned to you in the project. That is, the total count of all the sentences found in all the documents that have been assigned to you.
managers and superusers: The total number of samples
Labeled: (Visible to users with annotation assignments) How many of the samples assigned to you have you completed.
Precision, Recall, F1 Measure: These are only displayed once the model has started to learn. They are typically ready once about twenty samples in the training set have been annotated as well as some in the validation set.
Detailed view of annotations and status
Once you've created your data corpus and loaded the documents, at any point in the annotation process, you can review the corpus and the annotations. Select View Annotations to see a detailed view of your work.
All samples in the project are listed. The Filters panel allows you to filter the list by:
Annotator, selecting specific annotator(s) (manager only)
Annotation status (annotated, not annotated)
Adjudication status (adjudicated, not adjudicated, needs adjudication) (manager only)
Text, filtering for specific terms within the document text. The text field supports regular expression searches.
Document id
Set (train, eval)
Select on a column header to sort the results by that column. Shift-Select to add another column to the sort.
For each sample, you will see:
The original text, marked up with its annotations
The id of the source document
The annotators who have worked on this sample, and those who were assigned but have not yet completed the task (manager only)
The sample state (annotated or not annotated)
The sample’s dataset (training or evaluation)
If multiple annotators have been assigned to a sample’s document, you can see their status in the Annotators and Pending columns of the table. If annotators have created differing results, their work will be listed below the main result. You can click on any of these annotated texts to accept or adjudicate a result.
If there is a prediction for the sample, that will also be displayed.
Select Test Document on the action bar to enter short text documents and see how the model evaluates them.