Configuring Name Matching

Individual name tokens are scored by a number of algorithms or rules. These algorithms can be optimized by modifying configuration parameters, thus changing the final similarity score.

The parameter files are contained in two .yaml files located in plugins/rni/bt_root/rlpnc/data/etc. The parameters are defined in parameter_defs.yaml and modified in parameter_profiles.yaml.

parameterUniverseOne/spa_eng_PERSON:
    reorderPenalty: 0.4
    HMMUsageThreshold: 0.8
    stringDistanceThreshold: 0.1
    useEditDistanceTokenScorer: true
parameterUniverseOne/eng_eng:
    reorderPenalty: 0.6
    

curl -XPOST "http://localhost:9200/_search" -H 'Content-Type: application/json' -d'{ 
  "query": {
    "match": {
      "full_name": "A Ely Taylor"
    }
  },
  "rescore": {
    "window_size": 3,
    "rni_query": {
      "rescore_query": {
        "rni_function_score": {
          "name_score": {
            "field": "full_name",
            "query_name": "A Ely Taylor",
            "score_to_rescore_restriction": 1,
            "window_size_allowance": 0.5,
            "universe": "parameterUniverseOne"
          }
        }
      },
      "query_weight": 0,
      "rescore_query_weight": 1
    }
  }
}'

curl -XPOST "http://localhost:9200/_search" -H 'Content-Type: application/json' -d'{ 
  "query": {
    "match": {
      "full_name": "{ \"data\": \"A Ely Taylor\", \"universe\": \"parameterUniverseOne\"}"
    }
  },
  "rescore": {
    "window_size": 3,
    "rni_query": {
      "rescore_query": {
        "rni_function_score": {
          "name_score": {
            "field": "full_name",
            "query_name": "A Ely Taylor",
            "score_to_rescore_restriction": 1,
            "window_size_allowance": 0.5,
            "universe": "parameterUniverseOne"
          }
        }
      },
      "query_weight": 0,
      "rescore_query_weight": 1
    }
  }
}'

curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe" -H 'Content-Type: application/json' -d'{ 
  "profiles": [
    {
      "name": "parameterUniverseOne/spa_eng_PERSON",
      "parameters": {
        "reorderPenalty": 0.4,
        "HMMUsageThreshold": 0.8,
        "stringDistanceThreshold": 0.1,
        "useEditDistanceTokenScorer": true
      }
    }
  ]
}'

To start tuning the parameters, run the RNI pairwise match on the test set and look at the match reasons in the response. These match reasons will serve as a guide for which parameters to tune, which are defined in parameter_defs.yaml. For additional support on tuning the parameters, contact support@rosette.com.

Once you define a profile and set a parameter value, rerun the RNI pairwise match, scoring the match with the edited parameter_profiles.yaml file.

zho_zho_PERSON:  
  haniFourCornerCodeMismatchPenalty: 1

You can index null values and empty strings by updating the allowNullValue parameter. If the allowNullValue parameter is enabled, any document containing null values and empty strings for the fields rni_name, rni_address, and rni_date types will be successfully indexed, but search capabilities will be limited to valid values.

You can direct RNI to index documents with malformed strings of language by updating the ignoreBadData parameter. If the ignoreBadData parameter is enabled, any document containing a malformed language string will be successfully indexed, but search capabilities will be limited to valid languages.

By default these parameters are disabled. These features are useful when performing bulk operations in Elasticsearch.

The file name is parameter_profiles.yaml, located in plugins/rni/bt_root/rlpnc/data/etc/.

To turn any of these features on, set the value of the parameter ignoreBadData or allowNullValue in the above file to true.

To evaluate the newly tuned parameter values, query a large dataset of names or addresses that does not include your test set. For an exact evaluation, query an annotated dataset that includes the correct answers for a number of queries. For a general evaluation, measure the number of pair matches that have scores above your threshold, compared to before tuning the parameter values. If there were too many matches before, now there should be fewer matches. If there were too few matches before, there should be more now. If the number of matches increases or decreases dramatically, then there is a higher chance of missing correct matches below the threshold or including incorrect matches above the threshold.

If you find new pair matches that you want to score above or below your threshold, collect them into a test set to retune the parameters. Then evaluate the parameters again using a large dataset to review results. It is important to frequently evaluate new parameter settings on separate test data to ensure the parameters continue to return correct results.

Before running any matching algorithms, the names are transformed into tokens that can be compared. RNI uses stop patterns and stop word prefixes to remove patterns, including titles such as Mr., Senator, or General, that you do not want to include in name matching. Both stop patterns and stop word prefixes are used to strip matching name elements during indexing and querying. Stop words are string literals and are processed much more quickly than stop patterns, which are regular expressions. You should use stop words for the most efficient removal of prefixes, such as titles. Stop words are language-dependent.

For each name, RNI performs the following steps in order:

RNI cycles its way through the stop patterns then the stop words, each cycle removing the patterns and words that strip nothing, until the list of stop patterns and stop words is empty.

stopregexes_LANG[_TYPE].txt

^fnu\b
\blnu$ 

\blnu$    2
\blnu$    3

stopprefixes_LANG[_TYPE].txt 

You can create UTF-8 text files that specify the scores to be assigned for specified full-name pairs. The filename uses the ISO 639-3 three-letter language codes to designate the language of each full name in each of the full-name pairs:

fullnames_LANG1_LANG2[_TYPE].txt

where LANG1 is the three-letter language code for the first name and LANG2 is the three letter language code for the second name.

Each row in the file, except for rows that begin with #, is a tab-delimited full-name pair and score:

name1 Tab name2 Tab score

The scores must be between 0 and 1.0, where 0 indicates no match, and 1.0 indicates a perfect match.

The installation includes a sample file with sample entries commented out: plugins/rni/bt_root/rlpnc/data/rnm/ref/override/fullnames_eng_eng.txt. Any non-commented-out entries in this file assign scores to English queries applied to English names in an RNI index. For example,

John Doe    Joe Bloggs      1.0

indicates that the query name John Doe matches the index name Joe Bloggs (both used in different regions to indicate 'person unknown') with a score of 1.0.

These match patterns are commutative. The previous entry also specifies a match score of 1.0 if the query name is Joe Bloggs and the index includes a document with an rni_name field containing John Doe.

You can add entries for English to English name matches to fullnames_eng_eng.txt, and create additional override files, using the filename to specify the languages. For example the following entries could appear in fullnames_jpn_eng.txt:

外山恒         Toyama Koichi    1.0
ヒラリークリントン    Hillary Clinton    1.0

You can create text files that specify token (name-element) pairs that match. Token pair overrides are supported^[4] for English-English, Japanese-English, Chinese-English, Russian-English, Spanish-English, Japanese-Japanese, Russian-Russian, English-Korean, Korean-Korean, Spanish-Spanish, Greek-English and Hungarian-English token pairs. Such pairs may include proper name and nickname, such as Peter and Pete, and cognate names such as Peter and Pedro. Tokens cannot contain whitespace. When RNI evaluates two names, each of which contains an element from the pair, it enhances the value of the resulting name match score. For example, if Abigail and Abby constitute a token pair, then the match score for Abigail Harris and Abby Harris will be higher than it would be if the token pair had not been specified.

The token pairs may be within a language or cross-lingual, as indicated by the file name:

tokens_LANG1_LANG2_[TYPE].txt

where LANG1 is the three-letter language code for the first token in each pair and LANG2 is the three-letter language code for the second token in each pair. Each entry in the file, except for rows that begin with #, is a tab-delimited token pair and may include a raw score between 0.0 and 1.0 or an indicator that at least one of the tokens is a nickname or that the tokens are cognates:

Token1 Tab Token2 Tab [[0.0-1.0]|NICKNAME|COGNATE|VARIANT]

A token pair override score (raw score or indicator) serves as a minimum score, but you can write "/force" after a token score to force it to be exactly that value:

Token1 Tab Token2 Tab [([0.0-1.0]|NICKNAME|COGNATE|VARIANT)/force]

If you would like to prevent a token pair from matching, you can use the SUPPRESS indicator as an alias for "0.0/force". If you do not include NICKNAME, COGNATE, VARIANT, or SUPPRESS, RNI assumes NICKNAME.

RNI includes plugins/rni/bt_root/rlpnc/data/rnm/ref/override/tokens_eng_eng.txt, which contains a list of English/English token pairs. For example:

Peter    Pete    NICKNAME
Peter    Pedro   COGNATE

This directory also contains Chinese to English token overrides for LOCATION and ORGANIZATION: tokens_zho_eng_LOCATION.txt, tokens_zho_eng_ORGANIZATION.txt.

When you create an additional file in the same location, use the ISO 639-3 three-letter language name in the filename to identify the language of each name element in the pair. For example tokens_eng_eng.txt indicates that the contents match English names to English names; tokens_eng_eng_ORGANIZATION.txt indicates that the contents match English ORGANIZATION names to English ORGANIZATION names. The SDK includes a sample file for matching English/English tokens in LOCATION entities: tokens_eng_eng_LOCATION.txt.

We recommend that you enter the language names in alphabetical order in the filename and token pairs. Keep in mind that the order has no influence on the resulting score, since the scoring is commutative.

You can create text files that specify the normalized form for tokens (name elements) and variants to normalize to that form. The file name indicates the language and optionally the entity type for the tokens to be normalized:

equivalenceclasses_LANG_[TYPE].txt

For example, equivalenceclasses_jpn.txt would contain entries for normalizing Japanese token variants for any entity type to a normalized form.

Each entry in the file contains a normalized form followed by one or more variant forms. The syntax is as follows:

[normal_form1]
variant1_1
variant1_2
variant1_3
[normal_form2]
variant2_1
variant2_2
variant2_3
...

RNI includes plugins/rni/bt_root/rlpnc/data/rnm/ref/override/equivalenceclasses_eng_PERSON.txt, which contains a list of variant renderings to normalize to muhammad:

[muhammad]
mohammed
mahamed
mohamed
mohamad
mohammad
muhammed
muhamed
muhammet
muhamet
md
mohd
muhd

You can add lists of variants to this file, including the normalized form in square brackets to start each list.

You can edit the list of tokens that are given low influence in RNI. These low weight tokens are parts of a name (such as suffixes) that don't contribute much to the name matching accuracy.

The file name is lowWeightTokens_LANG.txt.

For example, plugins/rni/bt_root/rlpnc/data/rnm/ref/lowWeightTokens_eng.txt contains entries for tokens in English that you may want to put less emphasis on: "jr", "sr", "ii", "iii", "iv", "de".

Many companies have their own file of organizations with their different names. To improve matching between organization names, you can supplement the real world ids provided in RNI and build your own file of real world ids. The provided file will build a binary file in the specified output directory named <LANG>_ORGANIZATION_ids.bin where <LANG> is the three-letter language code of the file.

The input file is a tab separated file (.tsv). Each line contains an organization name and a corresponding alphanumeric id. The file can only contain a single language and script. You must create a separate file for each language.

IBM    WE1X92
Big Blue    WE1X92
International Business Machines    WE1X92

Unzip the file realWorldIDBuilder.zip found in the plugins/rni/bt_root directory and run the build command. Instructions on how to run the program are in the README.md file in the zip file.