Tokenizers

Type	Input	Tokenization (when option is enabled)	Tokenization (when option is disabled)	POS tag
emoji			Tokenization of emoji cannot be disabled.	EMO
emoticon	:)	:)	: )	EMO
@mention	@basistechnology	@basistechnology	@ basistechnology	ATMENTION
hashtag	#basistechnology	#basistechnology	# basistechnology	HASHTAG
email address	info@basistech.com	info@basistech.com	info @ basistech.com	EMAIL
URL	http://www.basistech.com	http://www.basistech.com	http : / / www.basistech.com	URL

The tokenization when the option is disabled depend on the options language and tokenizerType. The samples provided here are for when language=eng and tokenizerType=ICU.

Emoji are defined by Unicode Technical Standard #51. In tokenizing emoji, RBL recognizes the emoji presentation selector (VS16; U+FE0F) and text presentation selector (VS15; U+FE0E), which indicate if the preceding character should be treated as emoji or text.

Although RBL detects sideways, Western-style emoticons, it does not currently support Japanese-style emoticons called kaomoji such as (o^ ^o).

RBL normalizes emoji, placing the result into the lemma field. The simplest example is when an emoji presentation selector follows a character that is already an emoji. In this case, RBL will simply remove the emoji presentation selector.

Lemmatization applies to an emoji character in multiple ways.

Emoji that depict people or body parts may be followed by an emoji modifier indicating skin tone. Lemmatization simply removes the emoji modifier skin tone from the emoji character. The reasoning is that the skin tone is of secondary importance to the meaning of the emoji.

Surface form	Lemmatized form
( Boy + Medium Skin Tone)

Emoji depicting people may be followed by an emoji component indicating hair color or style. Lemmatizing removes the hair component from the emoji character.

Surface form	Lemmatized form
( Adult + ZWJ + Red Hair)

Where a gender symbol has been added to create a gendered occupation emoji, lemmatization removes the gender symbol.

Surface form	Lemmatized form
( Police Officer + ZWJ + ♀ Female Sign + VS16)

Finally, RBL can normalize non-fully-qualified emoji ZWJ sequences to fully-qualified emoji ZWJ sequences. In the above example, it is possible to omit the VS16 (though discouraged by Unicode): since Police Officer is an emoji, anything joined to it by a ZWJ is implicitly an emoji too. RBL adds the missing VS16.

A tokenization rule file is a ICU break rule file encoded in UTF-8. A custom file replaces Basis’s tokenization rules, so a custom rule should include all the rules for basic tokenization as well as the new custom rules. The default rule files that RBL uses can be obtained by contacting Basis support, or you can copy the rule file from ICU.

RBL also provides the ability to pass in a subrule file if desired. This is for splitting tokens produced according to rules in the main file. The subrule file is a list of subrules, each of which is a number and a regex separated by a tab character. This number corresponds to the “rule status”^[3] of the main rule whose tokens the subrule splits. Each capturing group in the subrule regex corresponds to a token that will be produced by the tokenizer.

The rule file and the subrule file can be placed anywhere. In particular, they need not be placed anywhere within your RBL installation directory.

There is one Basis-specific extension, !!btinclude <filename>. This command tells the preprocessor to replace the !!btinclude line with the contents of the specified file. Relative paths are relative to the location of the file containing the !!btinclude line. Recursive inclusion is allowed.

The ICU tokenizer does not normally tokenize with an eye to emoticons, but perhaps that is important to your use case. You could make a copy of the default rule file and add the following.

...
$Smiley = [\:=][)}\]];

!!forward;
$Smiley;
...

For the input:

=)          

instead of the output:

=
)       

of two tokens with the Basis default rules you would get back one token:

=)