How do computers read text using a tokenizer?
In short: Transformer models cannot understand raw text directly. You will learn how a tokenizer acts as a bridge, converting human language into numerical data that a neural network can process.
Since neural networks operate on numbers rather than characters, the first step in a machine learning pipeline is converting text into a format the model can interpret. A tokenizer performs this essential task by breaking down input text into smaller units called tokens, which can be words, sub-subwords, or even punctuation marks.
The process involves three main actions: splitting the text into these tokens, assigning a unique integer to every token, and adding extra information the model might need. For example, the tokenizer can produce an 'attention_mask' alongside the 'input_ids' to help the model focus on the correct parts of the data. To ensure accuracy, you must use the exact same preprocessing settings that were used during the model's original training, which is typically achieved by fetching the specific configuration from a Model Hub.
Based on: Hugging Face LLM Course — Behind the pipeline · Licence: Apache License 2.0 · View source →
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Source: Hugging Face LLM Course: Behind the pipeline · Apache License 2.0
To cite this card, link to https://intellectplus.co.uk/app/technology-how-do-computers-read-text-using-a-tokenizer.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in AI & machine learning: practical · More in Technology · Latest · A–Z · All topics