Why are hyphens left as part of the token?

For RLP (native), see: How does tokenization of European languages work (RLP)?

For RBL-JE, see: How does tokenization of European languages work (RBL-JE)?

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.




Powered by Zendesk