Note
You must be registered as an adjudicator for the project.
Adjudication is the process of reviewing different annotations for a sample and determining which is correct. If multiple annotators have annotated a sample and there are no disagreements between the annotators, automatically marks the sample as adjudicated, accelerating the adjudication process.
Select View Annotations from the project dashboard.
-
Under Filters select Needs adjudication.
All samples which have been annotated by at least 2 annotators and have differences between annotations will be displayed.
Select a sample.
-
Select Adjudicate.
The annotated samples will be displayed.
Choose the correct sample and select Adjudicate.
The system will display the next sample to adjudicate until there are no more samples requiring adjudication. At any point you can go back to View Annotations to select another sample.
The annotations in Rosette Adaptation Studio are sent to the REX Training Server and provide the data for training a new model. Only adjudicated annotations are included in training. The system will auto-adjudicate samples, decreasing the manual adjudication effort.
The rules for auto adjudication are:
If only one annotator has annotated a sample, the work is assumed to be correct and adjudicated.
If multiple annotators have supplied identical annotations for a sample, the work is assumed to be correct and adjudicated.
If multiple annotators have supplied contradictory annotations for a sample, then it is marked as "needs adjudication". It is not supplied for training until it has been manually adjudicated.
The system only considers work that has been completed by annotators. If additional annotators have been assigned to work on a document, but they have not completed their work yet, the samples may still be marked as adjudicated.
The system is an interactive environment. At any time, additional work can be done on a sample (a new annotator annotates the sample, an existing annotator revises their work). The modified sample replaces any earlier versions of the sample. Once adjudicated (manually or auto-adjudicated), it will be sent to training with the new annotations.