How should I interpret confidence scores returned from language detection?

The value returned by com.basistech.rosette.dm.LanguageDetection.DetectionResult.getConfidence() in the SDK or from the "confidence" JSON value in Rosette is not a traditional confidence measure, but rather a score comparing the strength of results on the same document. One way to quantify the certainty of the top result is to take the ratio of its score to that of the second result. On a strong result, the difference will be roughly an order of magnitude.

For example, assume the results of a particular analysis are:

The ratio of the first score (English) to the second score (French) is 0.4165 / 0.0048 = 9.2894. At nearly an order of magnitude difference, this suggests high confidence in the result.

Contrast an analysis that returns the results:

The ratio of the first score (Spanish) to the second score (Catalan) is 0.02812 / 0.01524 = 1.8458. This suggests a much less confident result.

Normalized confidence scores

For some applications, you may want normalized confidence scores that can be compared across analyses. As a best practice for this case, we recommend summing the scores for the first 5 results, and normalizing them to add up to 1.0. The resulting figures can be interpreted as confidence scores. Note that this is still not the same as a probability or likelihood -- the likelihood that a result is correct is often much higher than its confidence score.

Take the first example above. If you normalize the results to add up to 1.0, you get:

While 72.8% doesn't sound all that confident, it's much higher than the 7.8% confidence of the second highest result. In practice, this indicates a near certainty that the answer is correct.

Contrast the second example:

Here the confidence of the first result, 36.6%, accurately conveys that the likelihood of an accurate result is far from certain.