tokenization
/ˌtoʊkənaɪˈzeɪʃən/
Breaking text into smaller units for processing
“Tokenization splits sentences into words or subword pieces.”
Origin: From Old French token (sign, symbol), from Old English tacn; -ization from Greek suffix
/ˌtoʊkənaɪˈzeɪʃən/
Breaking text into smaller units for processing
“Tokenization splits sentences into words or subword pieces.”
Origin: From Old French token (sign, symbol), from Old English tacn; -ization from Greek suffix