Note
Adjudication is not available for Events projects.
Note
You must be registered as an adjudicator for the project.
Adjudication is the process of reviewing annotations that have disagreements and determining which is correct. The samples with the correct annotations are considered part of the gold data and can be used for training. If multiple annotators have annotated a sample and there are no disagreements between the annotators, the sample is marked as auto-adjudicated, accelerating the annotation process.
Adjudication can be initiated via View Annotations or directly from the project navigation bar.
The adjudication screen consists of the sample being adjudicated, a table displaying agreeing and disagreeing annotations, and several actions:
-
Clear Annotations: Remove all labels you have applied to entities in the sample.
-
Undo: Undo your most recent annotation.
-
Skip for now: Proceed to another sample without adjudicating the current sample.
-
Previous: Return to the previous sample.
-
Adjudicate: Save any labels applied to entities in the sample as adjudications.
Adjudicate via View Annotations
-
Select View Annotations from the project navigation bar.
-
Navigate to the desired a sample.
-
Select Adjudicate. Adaptation Studio displays the annotated sample.
-
Select an entity with disagreeing annotations and apply the correct label.
-
Repeat step 4 until you have applied the correct label to all entities with disagreeing annotations.
-
Select Adjudicate.
After you adjudicate a sample, Adaptation Studio displays the next sample to adjudicate until there are no more samples requiring adjudication. At any point you can go back to View Annotations to select another sample.
Adjudicate via Project Navigation Bar
-
Select Adjudicate from the project navigation bar. Adaptation Studio displays an annotated sample.
-
Select an entity with disagreeing annotations and apply the correct label.
-
Repeat step 2 until you have applied the correct label to all entities with disagreeing annotations.
-
Select Adjudicate.
After you adjudicate a sample, Adaptation Studio displays the next sample to adjudicate until there are no more samples requiring adjudication.
The annotations in Rosette Adaptation Studio are sent to the REX Training Server and provide the data for training a new model. Only adjudicated annotations are included in training. The system will auto-adjudicate samples.
The rules for auto adjudication are:
-
If only one annotator has annotated a sample, the work is assumed to be correct and adjudicated.
-
If multiple annotators have supplied identical annotations for a sample, the work is assumed to be correct and adjudicated.
-
If multiple annotators have supplied contradictory annotations for a sample, then it is marked as "needs adjudication". It is not supplied for training until it has been manually adjudicated.
-
The system only considers work that has been completed by annotators. If additional annotators have been assigned to work on a document, but they have not completed their work yet, the samples may still be marked as adjudicated.
-
The system is an interactive environment. At any time, additional work can be done on a sample (a new annotator annotates the sample, an existing annotator revises their work). The modified sample replaces any earlier versions of the sample. Once adjudicated (manually or auto-adjudicated), it will be sent to training with the new annotations.