Articles in this section

Evaluate

Avatar
Jason Stevens
  • Updated
Follow

Select the Evaluate tab from the navigation bar to start an accuracy evaluation. This allows you to:

  • Calculate the accuracy of RNI using your gold data.

  • Test different parameter configurations to determine the best settings for your data.

  • Determine the best threshold value for your data.

Evaluate requires the following inputs:

  1. A file containing your gold data, which is a set of annotated name match pairs.

  2. A match configuration, which includes a set of configuration parameter values.

Evaluating Name Matching Accuracy

Matching records which include names is a challenging problem because name spellings can differ in so many ways, including simple misspellings, to nicknames, truncations, variable spaces (Mary Ellen, Maryellen), spelling variations, and names written in different languages. Nicknames have a strong cultural component. The terminology itself is problematic because matching implies two things that are the same or equal, but name matching is more about how similar two entities are. Once you have a measure of similarity, you may need additional rules or human analysis to determine if it is a match. It is important to understand these challenges when evaluating name accuracy.

Evaluation Data

Data used to measure accuracy should include a wide variety of phenomena that make name matching challenging, including misspellings, aliases or nicknames, initials, and non-Latin scripts. Applying organizational domain knowledge to curating name data that contains specific phenomena found in your real world cases is an ideal starting point for crafting this data set.

Your data for testing accuracy should contain labeled or annotated data. This is often called gold data, referring to the accuracy of the training set's classification for supervised learning techniques. For name matching, it is a list of name pairs, where each pair is labeled as a match or not a match. You can’t calculate accuracy without labeled data. Since assigning classification labels to data can be subjective, you should use multiple annotators on the same data set, determining positive and negative name matches. Establishing a set of annotation guidelines for scoring a classification is necessary, as it provides consistency when classifying the data.

Once you've collected and annotated your gold data, create an evaluation file of name pairs to be imported and used in evaluation.

Evaluation Data File

The evaluation data file is a .csv file of annotated name pairs (gold data). It should include both positive (the names are considered a match) and negative (the names are not considered a match) name pairs. All name pairs must be the same entity type.

Table 4. Evaluate Import File

Column Name

Description

Required?

Example

Name1

First name in the name comparison

Yes

John R. Smith

Name1_Lang

3 letter ISO 693-3 language code

No

eng

Name2

Second name in the name comparison

Yes

Smith John

Name2_Lang

3 letter ISO 693-3 language code

No

eng

Entity_Type

What type of name is this? Person, Organization, Location, Date, or Address

Yes

PERSON

Match

Does Name1 match Name2?

Yes

Y

The first row of the file is a header row, containing the column names of the fields in the file. Each column is separated by a comma; if a value is not provided, that field is left blank but the column must still be included.

Sample File - PERSON

NAME1,NAME1_LANG,NAME2,NAME2_LANG,ENTITY_TYPE,MATCH
Peter Harding,eng,Pete Harding,eng,PERSON,Y
Peter Harding,eng,Harding Peter,eng,PERSON,Y
Peter Harding,,Pete Michael Harding,eng,PERSON,Y
Peter Harding,eng,P. M. Harding,eng,PERSON,N
Peter Harding,eng,Pat Harding,,PERSON,N
Peter Harding,eng,P. B. Harding,eng,PERSON,N
Peter Harding,eng,Pietro Hardin,eng,PERSON,N

Sample File - Address

123 Fake Street Springfield MO,,123 Fake St Springfield IL,,ADDRESS,N
820 Forest Road,,820 Forrest Rd,,ADDRESS,Y

To upload an evaluation data file:

  1. Select the Evaluate tab from the navigation bar.

  2. Drag or browse for the desired evaluation data file. When it has finished uploading, it will appear in the file list.

Measuring Accuracy

Precision, recall, and F1 score are metrics used to evaluate NLP tools. Accuracy is measured as a combination of the three values.

  • Precision answers the question "of the answers you found, what percentage were correct?" Precision is sensitive to false positives; higher is more precise.

  • Recall answers the question "of all possible correct answers, what percentage did you find?" Recall is sensitive to false negatives; higher is better recall.

  • F1 measure is the harmonic mean of precision and recall. The F1 measure is sensitive to both false positives and false negatives; a higher value means better accuracy. It isn't quite an average of the two scores, as it penalizes the case where the precision or recall scores are far apart. For example, if the system finds 10 answers that are correct (high precision), but misses 1,000 correct answers (low recall), you wouldn't want the F1 measure to be misleadingly high.

Export Match Configuration For RNI-Elasticsearch

Important

The exported .yaml file contains a comment with the RNI-Elasticsearch plugin version number. Only use this configuration file with the indicated version of RNI-Elasticsearch.

  1. Select Configure from the navigation bar.

  2. Select Export in the Options column for the desired match configuration.

  3. Rename the downloaded file parameter_profiles.yaml.

  4. Navigate to RNI-ES in your file browser.

  5. Replace the existing parameter_profiles.yaml in plugins/rni/bt_root/rlpnc/data/etc with the version exported in step 2.

For more information on RNI-Elasticsearch parameter configuration files, see Optimizing Name Matching with Parameters.

Calculating Precision, Recall, and F1

Let's look at how precision, recall, and F1-score are calculated. We have a set of name comparisons. From this we can calculate:

  • TPs: True positives. Number of matching name pairs that were labeled a match.

  • FPs: False positives. Number of matching name pairs that were not labeled a match.

  • FNs: False negatives. Number of name pairs that did not match, that were labeled a match.

Pairs: Number of rows in the gold data file; the number of name pairs compared = TP + FP

Matches: Number of records that are marked as matches in the gold data file. = TP + FN

Precision is an indication of how many of the matches are correct. If there are 2 correct matches, but 6 were identified as matches, P = .33. The name pair is correctly matched 1/3 of the time. If there are no false positives, the precision is 1.

Equation 1. 


Recall is an indication of how many matches were found. If 2 pairs are identified as matches, but there are 4 pairs that are actual matches, R = .5. This means that RMS found the correct match 1/2 the time. If there are no false negatives, the recall is 1.

Equation 2. 


F1-score is the harmonic mean of precision and recall

Equation 3. 


A major benefit of RMS is that you can define a threshold for name matching, optimizing for what is most relevant to your use case. For name matching, consider the case where a query name is expected to match one and only one name in the index. If there are three names returned above the threshold, including the correct match, then one name is a true positive (TP), two names are false positives (FP), and there are no false negatives (FN). If the correct match is not returned above the threshold, the number of false negatives will be one.

In this example, where the correct match is returned along with two other matches:

  • The precision is 1/3, since there is one correct match returned with two other matches.

  • The recall is 1.0, since there are no false negatives.

New Evaluation

Creates a new evaluation to run RNI against a data set along with a selected configuration, helping you configure RNI for your environment.

Before you can run a new evaluation, you must:

  1. Select the Evaluate tab from the navigation bar if you are not already on the Accuracy Evaluations screen.

  2. Select New Evaluation in the Options column for the gold data file you want to use.

  3. Verify that you have the desired evaluation data file selected from the drop-down menu.

  4. Select a match configuration from the drop-down menu. You can use the default configuration.

  5. Select Start Evaluation.

View Evaluation Data

Select View Evaluations to display all evaluations performed for a given evaluation data file. Use the Display settings to view the evaluation results for the best threshold, or for a specific threshold. Once you have specified the threshold, you can download the evaluation results for that threshold. Use Threshold Report to visualize how the results change as the threshold increases or decreases.

view_evaluation.PNG

Threshold Report

A key feature of RNI is that it returns a normalized score between 0 and 1 indicating how similar two names are. This makes integrating RNI into existing workflows powerful. However, it leads to the inevitable question: “What threshold should I use?” Without quantitative error analysis, this question is difficult to answer. RMS determines the correct value by processing your evaluation data against various thresholds, calculating precision, recall, and F1 for each data type.

The threshold report graphs the precision, recall, and F1 for each threshold value and each data type. As you can see from the graph, the threshold can be adjusted to favor precision or recall. With a high threshold, false positives will be less likely, leading to a higher precision. With a low threshold, false negatives will be less likely, leading to a higher recall. For applications involving border checking, you may be inclined to favor recall to avoid allowing potential threats into a country. Conversely, for Know Your Customer(KYC) applications, leaning your threshold towards precision may be best to reduce false positives. Your choice of threshold should ultimately be based on an analysis of accuracy as a function of threshold for your particular data, as well as your business requirements.

The threshold report page contains a table listing the Precision, Recall, and F1 values for each data type, along with the threshold value, along with a threshold graph. The threshold is the value which produces the best F1 score.

Figure 1. Threshold Report Graph

Threshold Report Graph

From this page, you can download the following reports as .csv files:

  • Threshold summary, listing the TP, TN, FP, FN, precision, recall, and F-scores for each threshold and entity type.

  • Detailed match results, listing the RNI score for each pair, along with the result (TP, TN, FP, or FN) for each threshold value.

Related articles

Comments

0 comments

Article is closed for comments.





Powered by Zendesk