Appendix

Match phenomena

Match phenomena describe why token spans did or did not match. For example, some match phenomena, such as HMM_MATCH, occur when tokens are matched by a particular scorer. Others, such as DELETION, occur when tokens cannot be matched at all.

Name	Description	Example
CONFLICT	The tokens do not match.	"william" and "johnson" are a CONFLICT.
DELETION	The token is unmatched.	When comparing "Richard William Smith" with "Richard Smith", "william" would be considered a DELETION.
EMBEDDING_MATCH	The tokens are semantically similar as determined by word-embedding vectors.	When comparing "boston building company" and "boston construction company", "building" and "construction" are an EMBEDDING_MATCH.
FIELD_BLOCKED	This field cannot be matched because of a cross-field match involving the same field in the other name.	When comparing "Bob\|William\|Smith" with "William\|\|Smith", "bob" is a FIELD_BLOCKED since the cross-field william match prevents it from matching with its corresponding field.
FIELD_CONFLICT	When comparing two names that are divided into fields, these fields do not match.	When comparing "Richard\|William\|Smith" with "Richard\|Johnson\|Smith", "william" and "johnson" would be considered a FIELD_CONFLICT.
FIELD_DELETION	When comparing two names that are divided into fields, this field is unmatched.	When comparing "Richard\|William\|Smith" with "Richard\|\|Smith", "william" would be considered a FIELD_DELETION.
GIVEN_NAME_DELETION	When comparing two names that are divided into fields, the GIVEN_NAME field is unmatched.	When comparing "Richard\|William\|Smith" and "\|\|William\|Scott", "Richard" will be a GIVEN_NAME_DELETION if that field in both names is marked as a `Given_name` field.
HANI_ABBREVIATION	One Hani token appears to be an abbreviation of another Hani token.	"北京大学" and "北大" are a HANI_ABBREVIATION match.
HMM_MATCH	The tokens are similar but not identical, and the match was determined by a particular model (hidden Markov model). This is a type of fuzzy match.	"richard" and "richerd" are an HMM_MATCH.
INITIALISM	One token is a name and the other token is the initials of the words which make up the name.	"john fitzgerald kennedy" and "JFK" are an INITIALISM. "consumer value stores" and "CVS" are an INITIALISM.
INITIAL_MATCH	One token is the first initial of the other.	"w" and "william" are an INITIAL_MATCH.
LANGUAGE_SPECIFIC_MATCH	The match was determined by a language-specific matcher.	"laden" and "لادن" are a LANGUAGE_SPECIFIC_MATCH.
MATCH	The tokens are identical (after stop word elimination and normalization).	"john" and "john" are a MATCH.
NULL	The NULL phenomenon is only listed in this table for completeness. It is only used internally and will never be returned in the SpanMatch object.	N/A
OUT_OF_ORDER_DELETION	This unmatched token still leaves the remaining tokens out of order when it is removed.	When comparing "George Herbert Walker Bush" with "George Bush Walker", "herbert" would be considered an OUT_OF_ORDER_DELETION.
OVERRIDE	The tokens appear as a pair on the override list. This is often used for nicknames.	"john" and "jack" will be an OVERRIDE match if they appear as a pair on the override list.
OVERRIDE_WRAPPED_HMM	One token is similar, but not identical to, the other token's counterpart on the override list.	"john" and "jakk" will be an OVERRIDE_WRAPPED_HMM match if "john" and "jack" appear as a pair on the override list.
PREFIX_INITIAL	One token is an initial that matches a prefix in the other token. In practice, the PREFIX_INITIAL phenomenon is rare.	If the `initialsScore` parameter is set to 0.1, "E Silva" and "EduardoSil" will be a PREFIX_INITIAL match.
SEQ2SEQ_MATCH	The tokens are similar but not identical, and the match was determined by a particular model (neural seq2seq). This is a type of fuzzy match.	"バラック・オバマ" and "Barack Obama" will be a SEQ2SEQ_MATCH when that model is active.
STRING_SIMILARITY	The tokens are similar in string edit distance (number of insertions, deletions, and substitutions) but not similar enough to be a fuzzy match.	"akcd" and "xkcd" are a STRING_SIMILARITY match.
STUCK_INITIAL	One name appears to have an initial mistakenly attached to a preceding token.	"DavidK" and "David Keith" are a STUCK_INITIAL match.
SURNAME_DELETION	When comparing two names that are divided into fields, the SURNAME field is unmatched.	When comparing "Richard\|William\|Smith" and "Richard\|William\|\|", "Smith" will be a SURNAME_DELETION if that field in both names is marked as a `Surname` field.
TRAILING_PATRONYMIC_DELETION^[a]	The unmatched token is a patronymic which has been truncated in the other name.	When comparing "Faisal bin Fahd bin Abdullah" and "Faisal bin Fahd", "bin Abdullah" is considered a TRAILING_PATRONYMIC_DELETION.
TRUNCATED_EXACT_MATCH	The tokens are identical except that one has been slightly truncated.	"murgatroyd" and "murgatroy" are a TRUNCATED_EXACT_MATCH.
TRUNCATED_HMM_MATCH	The tokens are similar, but not identical, and one has been slightly truncated.	"gilpatrickz" and "gillpatrick" are a TRUNCATED_HMM_MATCH.
UNKNOWN_FIELD_MATCH	One of the tokens is part of an "unknown" field in a fielded name. The UNKNOWN_FIELD_MATCH phenomenon is rare and usually requires use of the Java API.	When comparing "Richard\|William\|Smith" with "Richard\|William\|Scott", if the first field is an "unknown" field, "richard" and "richard" would be considered an UNKNOWN_FIELD_MATCH.
^[a] Only applies to Latin script names of Arabic origin.

Parameters

This table lists the parameters that can be configured via paramater_profiles.yaml. When applicable, each parameter has been linked to the specific match phenomena that it impacts. You can find more information on each parameter in paramater_defs.yaml.

Table 11. Parameter impacts

Parameter name	Applies to	Impacts
addressCrossFieldScoreThreshold	Addresses	Can affect any kind of match
addressDeletionScore	Addresses	DELETION match phenomenon
addressDifferentGroupPenalty	Addresses	Can affect any kind of match
addressFinalBias	Addresses	All address match scores
addressJoinedTokenLimit	Addresses	Concatenation^[a]
addressOverrideDefaultScore	Addresses	OVERRIDE match phenomenon
addressOverrideTablePath	File locations	Internal engineering detail
addressReorderPenalty	Addresses	Reordering^[e]
addressSameGroupPenalty	Addresses	Can affect any kind of match
addressStopPatternsPath	File locations	Internal engineering detail
addressUnpairedFieldScore	Addresses	FIELD_DELETION match phenomenon
adjustOneSidedDeletionScores	All names	DELETION match phenomenon
allowNullValue	Elasticsearch	Elasticsearch setting
alternativePairsToCheck	All names	Can affect any kind of match where English transliterations are being used, and where there are multiple possible transliterations (e.g., Chinese/Japanese/Korean readings of Han names)
alternativeTimeProximityMatch	Dates	All date match scores
boostWeightAtBothEnds	All names	All name match scores
boostWeightAtLeftEnd	All names	All name match scores
boostWeightAtRightEnd	All names	All name match scores
caseSensitiveData	All names	INITIALISM match phenomenon
cityAddressFieldWeight	Addresses	Weighting
cityDistrictAddressFieldWeight	Addresses	Weighting
cognateOverrideScore	All names	OVERRIDE match phenomenon for tokens marked as COGNATE in the override file
conflictScore	All names	CONFLICT match phenomenon
conflictThreshold	All names	CONFLICT match phenomenon
countryAddressFieldWeight	Addresses	Weighting
countryRegionAddressFieldWeight	Addresses	Weighting
crossFieldInitialsPenalty	Fielded names	INITIAL_MATCH match phenomenon
crossFieldJoinInitialPenalty	Fielded names	Concatenation^[a]
crossFieldJoinPenalty	Fielded names	Concatenation^[a]
crossFieldMatchPenalty	Fielded names	Can affect any kind of match
crossLanguageGenderConflictPenalty	All names	Gender mismatch^[b]
dateFinalBias	Dates	All date match scores
dateOrdering	Dates	All date match scores
dayDistanceWeight	Dates	All date match scores
deletionScore	All names	DELETION match phenomenon
detectableLanguages	All names	Currently, you can only enable detection of Latin script as Turkish or Vietnamese.
editDistanceScoreBias	All names	Can affect any kind of match
enableDynamicConfigurationEndpoints	Elasticsearch	Elasticsearch setting
enablePromisingTermFiltering	Speed/Accuracy	Performance only
enableYueReadings	All names	Names written in Han script
entranceAddressFieldWeight	Addresses	Weighting
equivalenceClassesPath	File locations	Internal engineering detail
estimatedConflictOrDeletionScore	All names	Internal engineering detail
exactLatnMatchScore	All names	Token normalization
expensiveScorerJoinedTokenLimit	All names	Concatenation^[a]
fieldBlockedScore	Fielded names	OUT_OF_ORDER_DELETION match phenomenon
fieldConflictScore	Fielded names	CONFLICT match phenomenon
fieldDeletionScore	Fielded names	DELETION match phenomenon
finalBias	All names	All name match scores
frequencyRankBias	All names	Can affect any kind of match
genderConflictPenalty	All names	Gender mismatch^[b]
genderConflictPenaltyThreshold	All names	Gender mismatch^[b]
globalTokenCacheConfig	Speed/Accuracy	Performance only
globalTokenPairCacheConfig	Speed/Accuracy	Performance only
haniAbbreviationScore	All names	INITIALISM match phenomena in Han script
haniAbbreviationThreshold	All names	INITIALISM match phenomena in Han script
haniFourCornerCodeMismatchPenalty	All names	Names written in Han script
hmmScoreBias	All names	HMM_MATCH phenomenon
hmmScoreLimit	All names	HMM_MATCH phenomenon
houseAddressFieldWeight	Addresses	Weighting
houseNumberAddressFieldWeight	Addresses	Weighting
ignoreBadData	Elasticsearch	Elasticsearch setting
improveSingleDigitManipulationMatch	Dates	Date match scores containing exactly one instance of digit manipulation^[c] and no other differences
initialFrequencyRank	All names	INITIAL_MATCH match phenomenon
initialismMismatchPenalty	All names	HMM_MATCH phenomenon
initialismScore	All names	INITIALISM match phenomenon
initialsConflictScore	All names	CONFLICT match phenomenon
initialsDeletionPenalty	All names	DELETION match phenomenon
initialsScore	All names	INITIAL_MATCH match phenomenon
islandAddressFieldWeight	Addresses	Weighting
joinedTokenInitialsPenalty	All names	Concatenation^[a] INITIAL_MATCH match phenomenon
joinedTokenLimit	All names	Concatenation^[a]
joinedTokenPenalty	All names	Concatenation^[a]
levelAddressFieldWeight	Addresses	Weighting
libpostalDataDirPath	File locations	Internal engineering detail
lowWeightTokenFrequencyRank	All names	Can affect any kind of match
lowWeightTokenPath	File locations	Internal engineering detail
maximumAlternateTokenizationRelativeDistance	All names	Affects tokenization and therefore any potential score
maximumOrganizationInitialismLength	Organization names	INITIALISM match phenomenon
maximumPersonInitialismLength	Person names	INITIALISM match phenomenon
maxYearDistanceForDigitManipulation	Dates	Date match scores containing exactly one instance of digit manipulation^[c] and no other differences.
minFieldWeightFactor	Fielded names	Weighting
minimumAlternateTokenizationLength	All names	Affects tokenization and therefore any potential score
minimumOrganizationInitialismLength	Organization names	INITIALISM match phenomenon
minimumPersonInitialismLength	Person names	INITIALISM match phenomenon
monthDistanceWeight	Dates	All date match scores
nameBigramQueryBoost	Lucene searches	First-pass accuracy Can affect any kind of match
nameDoubleMetaphoneQueryBoost	Lucene searches	First-pass accuracy Can affect any kind of match
nameGluedQueryBoost	Lucene searches	First-pass accuracy Can affect any kind of match
nameInitialQueryBoost	Lucene searches	First-pass accuracy Can affect any kind of match
nameLengthMismatchPenalty	All names	DELETION match phenomenon Concatenation^[a] Any phenomenon that changes the number of tokens in a name
nameRealWorldIdQueryBoost	Lucene searches	First-pass accuracy Can affect any kind of match
ngramLMPath	File locations	Internal engineering detail
ngramThresholdPath	File locations	Internal engineering detail
nicknameOverrideScore	All names	OVERRIDE match phenomenon for tokens marked as NICKNAME in override file
numericTokenFrequencyRank	All names	Can affect any kind of match
outOfOrderDeletionScore	All names	OUT_OF_ORDER_DELETION match phenomenon
parseUnknownFieldMarker	Fielded names	UNKNOWN_FIELD_MATCH match phenomenon
poBoxAddressFieldWeight	Addresses	Weighting
postCodeAddressFieldWeight	Addresses	Weighting
queryAlternativeOriginLanguages	Speed/Accuracy	Can affect any kind of match
realWorldIdsPath	File locations	Internal engineering detail
realWorldIdsPathUser	File locations	Internal engineering detail
reorderCorrection	All names	Rotation ^[d]
reorderCorrectionThreshold	All names	Rotation^[d]
reorderPenalty	All names	Reordering^[e]
rniFullnameOverridesPath	File locations	Internal engineering detail
rntFullnameOverridesPath	File locations	Internal engineering detail
roadAddressFieldWeight	Addresses	Weighting
sameNameUnknownFieldMatchInterpolator	Fielded names	UNKNOWN_FIELD_MATCH match phenomenon
staircaseAddressFieldWeight	Addresses	Weighting
stateAddressFieldWeight	Addresses	Weighting
stateDistrictAddressFieldWeight	Addresses	Weighting
stopPatternsPath	File locations	Internal engineering detail
stringDistanceWeight	Dates	All date match scores
stuckInitialAffixMinLength	All names	STUCK_INITIAL match phenomenon
stuckInitialScore	All names	STUCK_INITIAL match phenomenon
suburbAddressFieldWeight	Addresses	Weighting
thresholdToDropoffBiasMapping	Dates	All date match scores
timeDistanceWeight	Dates	All date match scores
timeProximityYearInterval	Dates	All date match scores
tokenizeOrganizationsWithNumbers	Organization names	Affects tokenization and therefore any potential score
tokenOverridesPath	File locations	Internal engineering detail
trailingPatronymicDeletionScore	Person names	TRAILING_PATRONYMIC_DELETION match phenomenon
truncationFractionLimit	All names	TRUNCATED_EXACT_MATCH match phenomenon TRUNCATED_HMM_MATCH match phenomenon
truncationScorerBias	All names	TRUNCATED_EXACT_MATCH match phenomenon TRUNCATED_HMM_MATCH match phenomenon
tryAlternateTokenization	All names	Affects tokenization and therefore any potential score
tryDayMonthSwap	Dates	All date match scores
unigramLMPath	File locations	Internal engineering detail
unitAddressFieldWeight	Addresses	Weighting
unknownFieldFrequencyRank	Fielded names	UNKNOWN_FIELD_MATCH match phenomenon
unknownVsKnownScore	Fielded names	UNKNOWN_FIELD_MATCH match phenomenon
unknownVsUnknownScore	Fielded names	UNKNOWN_FIELD_MATCH match phenomenon
useEmbeddings	Organization names	EMBEDDING_MATCH match phenomenon
useSolrPhraseQueries	Solr	Solr plugin setting
variantOverrideScore	All names	OVERRIDE match phenomenon for tokens marked as VARIANT in the override file
worldRegionAddressFieldWeight	Addresses	Weighting
yearDistanceWeight	Dates	All date match scores
^[a] Concatenation occurs when adjacent tokens are joined together to see if the resulting compound token will be a good match for any tokens in the other name. ^[b] Gender mismatch occurs when the apparent genders of the two names being compared do not match. ^[c] A digit manipulation is a transformation to a digit that can be accomplished with minimal additional lines. Digit manipulations may be intentional or the result of an OCR error. Our list of possible digit manipulations includes: 0↔8, 1↔7, 3↔8, 5↔8, 5↔6, 6↔8, 7↔2. ^[d] If the tokens in a name have been rotated, the reorder penalty will negatively impact the match score. RNI detects and compensates for this error. ^[e] Tokens that match, but that appear to be out-of-order, have their match scores adjusted to reflect that fact.

Internal parameters

This table lists the parameters that can be configured via internal_param_profiles.yaml. When applicable, each parameter has been linked to the specific match phenomena that it impacts. You can find more information on each parameter in internal_param_defs.yaml.

Important

We recommend against modifying these parameters unless advised to by Rosette support.

Name	Applies to	Impacts
affixGlueThreshold^[a]	All names and addresses	Concatenation^[b]
allLanguageSupport	All names	Can affect any kind of match
allowCacheBonuses	All names	Internal engineering detail
alwaysComputeSuffixes^[a]	All names	TRUNCATED_EXACT_MATCH match phenomenon TRUNCATED_HMM_MATCH match phenomenon
araRNISpeedOption	Translated names	Speed and accuracy tradeoff
crossSurnameMatchPenalty	All names	Matches in languages with onomastic information
debuggableIndex	N/A	Internal engineering detail Has no effect on matching
debugPrintTuples	N/A	Internal engineering detail Has no effect on matching
defaultScoreToCheckRestriction	All names Dates Addresses	First-pass scoring
disabledLanguages	All names
doFrontTruncations	All names	TRUNCATED_EXACT_MATCH match phenomenon TRUNCATED_HMM_MATCH match phenomenon
doQueryBigrams	All names	First-pass accuracy
doQueryCompleted	All names	First-pass accuracy
doQueryFullnameOverrides	All names	First-pass accuracy
doQueryFuzzy	All names	First-pass accuracy
doQueryGlued	All names	First-pass accuracy
doQueryIndexKeys	All names	First-pass accuracy
doQueryInitials	All names	First-pass accuracy
doQueryNormalized	All names	First-pass accuracy
doQueryPersonInitialisms	All names	First-pass accuracy
doQueryPhrase	All names	First-pass accuracy
doQueryRealWorldIds	All names	First-pass accuracy
doQueryTokenOverrides	All names	First-pass accuracy
doQueryTranslated	All names	First-pass accuracy
doViterbiRescaling	All names	Fuzzy match^[c]
editDistanceTokenScorerPenalty	All names	STRING_SIMILARITY match phenomenon
embeddingBias	Organization names	EMBEDDING_MATCH match phenomenon
embeddingZeroScore	Organization names	EMBEDDING_MATCH match phenomenon
enableAdditionalOnomastics	All names	Matches in languages with onomastic information
enableRemoteTokenScorer	All names	Japanese/English fuzzy match^[c]
enableSeq2SeqTokenScorer	All names	Japanese/English fuzzy match^[c]
enableTokenPairLogging	N/A	Internal engineering detail Has no effect on matching
engEngFastMode	All names	English/English fuzzy match^[c]
expandedLanguages	All names	Fuzzy match^[c] OVERRIDE match phenomenon
expansionLimit	All names	Fuzzy match^[c] OVERRIDE match phenomenon
expansionScoreThreshold	All names	Fuzzy match^[c] OVERRIDE match phenomenon
familiarTokenMismatchPenalty	All names	Can affect any kind of match
familiarTokenThreshold	All names	Can affect any kind of match
firstPassDayRange	Dates	Performance only
firstPassMonthRange	Dates	Performance only
firstPassYearRange	Dates	Performance only
foreignAddressFinalBias	Addresses	All English-to-non-English address matches
genderPenaltyMinimumLength	All names	Gender mismatch^[d]
givenFieldDeletionScore	Fielded names	DELETION match phenomenon
HMMCachePerProcess	All names	Internal engineering detail HMM_MATCH phenomenon
HMMCachePerThread	All names	Internal engineering detail HMM_MATCH phenomenon
hmmNormBias	All names	Internal engineering detail Fuzzy match^[c]
HMMUsageThreshold	All names	Internal engineering detail HMM_MATCH phenomenon
identifierEditDistanceTokenScorerPenalty	Identifiers	STRING_SIMILARITY match phenomenon
ignoreTranslationOrigins	All names	Can affect any kind of match that uses English transliteration
includeExtraKatakanaPersonReadings	Translated names	Can affect any kind of match
initialAndSuffixMinLength	All names	Fuzzy match^[c] INITIAL_MATCH match phenomenon
initialAndSuffixScore	All names	Fuzzy match^[c] INITIAL_MATCH match phenomenon
jniBias	All names	Can affect any kind of match in languages that use a JNI scorer
jpnRNISpeedOption	Translated names	Speed and accuracy tradeoff
kanjiMismatchPenalty	All names	Normalization of tokens that include kanji
katakanaTransliterationsOnly	Translated names	Can affect any kind of match
korRNISpeedOption	Translated names	Speed and accuracy tradeoff
latinDataAlternativesToCheck	All names	Can affect any kind of match where English transliterations are being used, and where there are multiple possible transliterations (e.g., Chinese/Japanese/Korean readings of Han names)
limitedLanguageEditDistance	All names	STRING_SIMILARITY match phenomenon
maxIdentifierEditDistance	All names	First-pass accuracy
notExactMatchPenalty	All names	Normalization
_postCodePathAddressFieldWeight	Addresses	Weighting
promisingFuzzyTermFrequencyFactor	Speed/Accuracy	Performance only
promisingTermFrequencyFactor	Speed/Accuracy	Performance only
queryMaxResults	All names Dates Addresses	First-pass scoring
queryMaxToCheck	All names Dates Addresses	First-pass scoring
queryMaxToConsider	All names Dates Addresses	First-pass scoring
queryToCheckAllowance	All names Dates Addresses	First-pass scoring
realWorldIdScore	Organization names	Real-world id match^[e]
remoteTokenScorerURL	All names	Internal engineering detail Japanese/English fuzzy match^[c]
rntTokenOverridesPath	File locations	Internal engineering detail
rusRNISpeedOption	Translated names	Speed and accuracy tradeoff
secondarySurnameTokenTypeWeight	All names	Matches in languages with onomastic information
seq2seqCachePerProcess	All names	Internal engineering detail Japanese/English fuzzy match^[c]
seq2seqCachePerThread	All names	Internal engineering detail Japanese/English fuzzy match^[c]
seq2seqTokenOverridesPath	File locations	Internal engineering detail
seq2seqUsageThreshold	All names	Internal engineering detail Japanese/English fuzzy match^[c]
splitTokens	All names	Internal engineering detail
stringDistanceThreshold^[a]	All names	Fuzzy match^[c]
surnameFieldDeletionScore	fielded names	DELETION match phenomenon
surnameTokenTypeWeight	All names	Matches in languages with onomastic information
taggerMinimumConfidenceThreshold	All names	Matches in languages with onomastic information
translatorResultsToKeep	translated names	Can affect any kind of match
truncationAffixSimilarityLength	All names	TRUNCATED_EXACT_MATCH match phenomenon TRUNCATED_HMM_MATCH match phenomenon
truncationAffixSimilarityThreshold	All names	TRUNCATED_EXACT_MATCH match phenomenon TRUNCATED_HMM_MATCH match phenomenon
truncationLengthLimit	All names	TRUNCATED_EXACT_MATCH match phenomenon TRUNCATED_HMM_MATCH match phenomenon
useCharacterLM	All names	Can affect any kind of match
useEditDistanceTokenScorer	All names	STRING_SIMILARITY match phenomenon
useIdentifierEditDistanceTokenScorer	Identifiers	STRING_SIMILARITY match phenomenon
useLM	All names	Can affect any kind of match
useOldAndNewNameSegmentationForJapanese	All names	Can affect any kind of match involving Japanese translations
useRealWorldIds	all names (or just orgs?)	Real-world id match^[e]
zhoRNISpeedOption	Translated names	Speed and accuracy tradeoff
^[a] Unlike public parameters for this feature, this is a speed/accuracy tradeoff, not a science-tuning parameter. ^[b] Concatenation occurs when adjacent tokens are joined together to see if the resulting compound token will be a good match for any tokens in the other name. ^[c] A fuzzy match is a match between tokens that are similar but not identical. The HMM_MATCH and SEQ2SEQ_MATCH phenomena are examples of this. ^[d] Gender mismatch occurs when the apparent genders of the two names being compared do not match. ^[e] RNI contains real world identifiers for corporations, which pair an entity id with nicknames and common permutations of the corporation name

Articles in this section

Match phenomena

Parameters

Internal parameters

Important

Comments

Articles in this section

Match phenomena

Parameters

Internal parameters

Important

Related articles