RNI can match dates returning a data match score reflecting the time similarity of the two dates. Dates that are closer together are considered a stronger match and return a match score closer to 1.
For example, 11/05/1993 and 11/07/1993 have a high score, as they are very similar and just two days apart. However, 11/05/1993 and 11/05/1995 yield a low score as they differ by two years.
The process is similar to name matching:
Index the dates in connection to the related names.
Query the date and name, receiving back a match score.
The query will return separate match scores for the name and for the associated date of birth. You may decide that the name is more important than the birth date. Within your system, you can weight and combine the name and date match scores to determine the final match score.
A date contains a year, month, and day, but not all fields are required for matching. All common delimiters for English dates are supported and dates can be expressed with various orderings. RNI will filter out some non-date related words. Formats that include time of day are not supported unless you specify an Elasticsearch date format that includes time information in the mapping. The time component will be ignored.
Omit fields if you do not have the value for one or more fields. For example: 1955-12-30, 1955--03, 12/30, -12-, --30, 1955, 1955-12-.
RNI supports a wide variety of date formats.
Days can be represented by 1 or 2 digits
Months can entered as numerics (1 or 2 digits) or English characters (full name or 3 character abbreviation)
Years can be represented as 1, 2, 3 or 4 digits
Supported delimiters include , . - /, as well as a space
Partial fields can be entered
Examples: All of the following are acceptable:
3/24/1984
March 1984
3-24
-24-84
March 24, 1984
24-3-84
March
1984
-
Create an index.
curl -XPUT 'http://localhost:9200/rni-test'
-
Define a mapping for fields that will contain dates. The type for a date field when matching is "rni_date".
curl -XPUT 'http://localhost:9200/rni-test/_mapping' -H'Content-Type: application/json' -d '{
"properties" : {
"birth_date" : { "type" : "rni_date" },
"primary_name" : { "type" : "rni_name" }
}
}'
Optionally, in the mapping, you can specify an Elasticsearch date format. All dates must adhere to the specified format. If you specify a format that includes time information, RNI ignores the time component of the date.
Warning
Specifying an Elasticsearch format disables support for unspecified fields. If, for example, you select a format that does not include a day field ("MM-yyyy"), you will get an error when you use the date format in a query.
curl -XPUT 'http://localhost:9200/rni-test/_mapping' -H'Content-Type: application/json' -d '{
"properties" : {
"birth_date" : {
"type" : "rni_date",
"format" : "MM-yyyy-dd"
},
"primary_name" : {
"type" : "rni_name"
}
}
}'
-
Index documents containing a date field.
curl -XPUT 'http://localhost:9200/rni-test/_doc/1' -H'Content-Type: application/json' -d '{
"primary_name" : "Joe Schmoe",
"birth_date" : "07-1955-24"
}'
There are many ways to incorporate date matching within your query. Here are two examples, one with date matching by itself, and one with date and name matching.
Base Query. The base query is a standard query against the date field. Refer to Query the Index.
curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
"query" : {
"match" : {
"birth_date" : "08-1955-25"
}
}
}'
RNI Rescore with Dates. Refer to Rescoring with RNI Pairwise Name Match.
curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
"query" : {
"match" : { "birth_date" : "08-1955-25" }
},
"rescore" : {
"query" : {
"rescore_query" : {
"function_score" : {
"date_score" : {
"field" : "birth_date",
"query_date" : "08-1955-25"
}
}
},
"query_weight" : 0
}
}
}'
The query returns a hit, with the RNI date match score.
"hits": {
"total": 1,
"max_score": 1.618923,
"hits": [
{
"_index": "test",
"_type": "_doc",
"_id": "AVXMepnorGuybmuiQtQr",
"_score": 0.8120856,
"_source": {
"primary_name": "Joe Schmoe",
"birth_date": "07-1955-24"
}
}
]
}
Base Query. The base query is a standard query against the date and name fields.
curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
"query": {
"bool": {
"should": [
{
"match": {
"primary_name": "Joe S."
}
},
{
"match": {
"birth_date": "08-1955-25"
}
}
]
}
}'
RNI Rescore with Dates. Use the doc_score function in the rescore when matching a combination of Elasticsearch field types instead of the functions for a single type (name_score and date_score). The name field is also added to the rescore.
curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
"rescore": {
"query": {
"rescore_query": {
"function_score": {
"doc_score": {
"fields": {
"primary_name": {
"query_value": "Joe S."
},
"birth_date": {
"query_value": "08-1955-25"
}
}
}
}
},
"query_weight": 0
}
}
}'
Similarly to the name matching parameters, there are a series of date matching parameters. The parameter values can be edited in the plugins/rni/bt_root/rlpnc/data/etc/parameter_defs.yaml file.
tryDayMonthSwap is on by default. This parameter attempts to correct for parsing errors by swapping the day and month. Turn it off if you only want to match the dates exactly as indexed.
Because dates are sometimes written month day and other times written day month, swap tries matching the date fields as written as well as with the month and date fields switched. The best score is returned as the match score. For example, if the dates in question are 1970-3-5 and 1970-6-4, this feature will match the following four pairs:
The date weighting fields control the relative strength of each aspect of the date-matching algorithm. A separate score is calculated for each match type. The final match score is calculated by performing a weighted arithmetic mean over each of the similarity scores. If a field is missing from a record, that field is ignored and its weight evenly distributed across other fields.
Table 1. Date Weighting Parameters
Display Name |
Parameter Name |
Score based on |
Example |
Time Weight |
timeDistanceWeight |
The number of days in between the two input dates |
1979-12-31 and 1980-1-1 look different, but their time difference is very close. They will have a high match score. |
Year Weight |
yearDistanceWeight |
The difference of the year fields |
Close years will have a high match score. |
Month Weight |
monthDistanceWeight |
The difference of the month fields |
1 and 12 are far, even if they are close in time. They will have a low match score. |
Day Weight |
dayDistanceWeight |
The difference of the day fields |
1 and 30 are far, even if they are close in time. They will have a low match score. |
String Weight |
stringDistanceWeight |
The edit difference between the two dates, when converted to a standard string (05021974 for 5/2/1974) |
1979-12-31 and 1980-1-1 will be 19791231 and 198000101. They will have a low match score. |
Dates with a high Time match score may have a very low String match score. Time finds date that are close together; String gives high scores to similarly formatted dates.