How should I interpret confidence scores returned from language detection?

The value returned by com.basistech.rosette.dm.LanguageDetection.DetectionResult.getConfidence() in the SDK or from the "confidence" JSON value in the Rosette API is not a traditional confidence measure, nor can it be compared across RLI results on different documents. Rather, it is a score comparing the strength of two results on the same document. For a strong result, the score will be roughly an order of magnitude higher than the score on the second-highest result -- even though both scores may seem ridiculously small.

As a best practice, we recommend summing the scores for the first 5 results, and normalizing them to add up to 1.0. The resulting figures can be interpreted as confidence scores, although not likelihoods -- just because the confidence score is 75% doesn't mean the likelihood of a correct answer is 75%. Usually the likelihood is much higher.

For example, the results of a particular analysis might be:

name iso score
English eng 0.04165
French fra 0.00448
Norwegian nor 0.00380
Romanian ron 0.00365
Dutch nld 0.00362

If you normalize the results to add up to 1.0, you get:

name iso score confidence
English eng 0.04165 0.7281
French fra 0.00448 0.0784
Norwegian nor 0.00380 0.0664
Romanian ron 0.00365 0.0638
Dutch nld 0.00362 0.0633

While 72.8% doesn't sound all that confident, it's much higher than the 7.8% confidence of the second highest result. In practice, this indicates a near certainty that the answer is correct.

Contrast an analysis that returns the results:

name iso score confidence
Spanish spa 0.02812 0.3657
Catalan cat 0.01524 0.1981
Portuguese por 0.01389 0.1806
Romanian ron 0.01032 0.1341
French fra 0.00934 0.1215

Here the confidence of the first result, 36.6%, is not dramatically better than that of the second result, 19.8%. The likelihood that the text is truly Spanish is far from certain.

Another way to quantify the certainty of the top result is to take the ratio of its score to that of the second result. In the English example above, the ratio is 0.4165 / 0.0048 = 9.2894. The ratio in the Spanish example is 0.02812 / 0.01524 = 1.8458. These numbers are, indeed, comparable, making it intuitively obvious that the English result is far more certain than the Spanish result.

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.




Powered by Zendesk