Skip to main content
Glite Dictionary
✂️ To split text 🪙 To make digital tokens 🔐 To hide information
to tokenize

to tokenize

7.0
To break writing into words or symbols so a computer can use it
  • verb
  • /ˈtoʊkəˌnaɪz/
  • Specialized
translation icon : tokenizar
  • I assigned numerical values to your questions by tokenizing them in an embedding matrix and generating a predictive model for the most effective answers, which I provided without lying.

It needs to tokenize, parse and annotate and retrieve an intelligent result with a high confidence.

Examples

  • Before we can process the text, we need to tokenize it to identify each separate token.

  • It needs to tokenize, parse and annotate and retrieve an intelligent result with a high confidence.

    Stanford

  • The software will tokenize the sentences into individual words for easier analysis.

  • In natural language processing, the first step is to tokenize the input data into recognizable units.

Synonyms

to explode
vsto tokenize
  • Specialized
1 6.3

To split text into parts at a mark like a comma

is a broader breaking into words or symbols not limited to delimiter splitting
to token
vsto tokenize
  • Specialized
7.5

To split text or data into separate parts for analysis

focuses on splitting text into words, symbols, or small items for computing

How Detailed

to tokenize
  • Specialized
7.0

Surface Forms

tokenize infinitive
tokenized past
tokenized past participle
tokenizing present participle
tokenizes third person singular

Morphology

tokenize = token (transparent) = token + ize

Formed by the productive verb-forming suffix '-ize' added to the noun 'token', producing the straightforward meaning 'make into tokens'.

Etymology

Tokenize comes from the simple word token and the ending -ize, where token once meant a small 'sign' or a 'ticket' and -ize means 'to make'. So to tokenize text is like giving each word its own little ticket or unit so a computer can handle it.