to token
- verb
- /ˈtoʊkən/
- Specialized
:
tokenizar
Examples
-
So I say which word I token that is depends on which word was selected from my mental lexicon.
-
Love tokens.
-
While from a positive standpoint, the ephemeral existence of these publications could be taken as evidence for the weakening of support for reformist literature, in a less positive view, they tokened continuing resistance by innovators.
Academic text (2010) -
To analyze the text effectively, we need to token it into smaller segments for better understanding.
-
When processing data, programmers often choose to token the input for further analysis.
-
The first step in natural language processing is to token the sentences into individual words.
Synonyms
12
3.5
To divide something into separate parts
11
4.8
To break text or data into parts so a computer can read it
To break writing into words or symbols so a computer can use it
Idioms
Compounds
1
4.0
A very small amount of money given to show you will pay later or to keep an agreement
- Texting
A digital system that gives tokens for actions and lets users trade or spend them for services
- Specialized
A system that gives tokens for good behavior and lets people exchange them for rewards
Surface Forms
Etymology
Token comes from the noun token, which goes back to an old English word tacn meaning 'sign' or 'mark'. To token text is to cut it into small 'units' or 'signs' so a program can work with them.