tokenization

Breaking text into smaller units for processing

“Tokenization splits sentences into words or subword pieces.”

Etymology — From Old French `token` (sign, symbol), from Old English `tacn`; `-ization` from Greek suffix