Token
A chunk of text, such as a word or part of a word, that an AI model reads and generates as its basic unit.
In plain English
A token is a small piece of text, like a word or part of a word, that an AI reads one at a time. Models count tokens to measure how much text they can handle.
Technical definition
A token is the atomic unit produced by a tokenizer that splits raw text into subword, word, or character pieces mapped to integer IDs. Language models operate over sequences of tokens, and their context window and billing are typically measured in token counts.
Business use case
Understanding tokens helps businesses estimate and control the cost of AI features, since most providers charge per token processed. It also guides how much content can be sent to a model at once, informing design choices like summarization or chunking.
Example
The word 'unbelievable' might be split into the tokens 'un,' 'believ,' and 'able,' which counts as three tokens rather than one word.
Frequently asked questions
A token is the basic unit of text a language model processes; it can be a whole word, part of a word, or a punctuation mark, depending on how the text is split.
Models have limits on how many tokens they can handle at once, and pricing is often based on tokens, so understanding them helps manage cost and fit content within context limits.
On average, one token is roughly three-quarters of an English word, so 100 tokens equal about 75 words, though this varies by language and text.
Keep exploring
Large Language Model
A large language model is an AI trained on huge amounts of text so it can read your question and write a useful answer. It powers chatbots and writing assistants.
Natural Language Processing
Natural language processing is the part of AI that helps computers understand and use human language. It is what lets software read your text, translate it, or figure out how you feel.
Embeddings
Embeddings turn words, sentences, or images into lists of numbers that capture their meaning. Things with similar meaning get similar numbers, so a computer can tell what is related.
Put AI intelligence to work in your business
Sitebard AI brings together the data, guides, and career intelligence you need to make confident AI decisions.