T5
T5, short for Text-to-Text Transfer Transformer, is a Machine learning model developed by Google Research that made a deceptively simple proposal: what if every problem in natural language processing could be phrased the same way? Authors create a framework to convert any NLP problem into a natural language format and use it to compare strategies to apply LLMs to NLP tasks, and the resulting model is called T5, short for "Text-to-Text Transfer Transformer". Translation, summarization, classification, question answering — all of it becomes a matter of feeding in text and reading out text, with a task-defining prefix doing the routing. For example, translating a sentence from English to French is formatted as "translate English to French: [input sentence]", while summarizing a paragraph is formatted as "summarize: [input text]".
Under the hood, T5 is a faithful descendant of the original Transformer architecture from Attention Is All You Need, keeping the full encoder-decoder shape rather than stripping it down to one half, as encoder-only BERT and decoder-only models like GPT-3 do. The encoder-decoder architecture of T5 is especially suited for sequence-to-sequence tasks, which cannot be performed by encoder-only models like BERT and can prove to be challenging for decoder-only models like GPTs due to the lack of explicit conditioning on source context. This lineage traces back through Sequence to sequence learning and the broader tradition of Neural machine translation, with Self-attention doing the heavy lifting of relating words to one another across long spans of text — a descendant of ideas about Attention that once had to be bolted onto Long short-term memory networks.
T5's pretraining trick is "span corruption": chunks of a document are masked out and the model must reconstruct them, a cousin of BERT's masked-word game but adapted for generative output. The encoder of T5 uses a span masking task, in which a number of tokens are replaced by a singular mask token, while the decoder uses masked self-attention in an autoregressive manner to predict the spans. Fueling this training was the Colossal Clean Crawled Corpus (C4), a scrubbed-down slice of the open web released alongside the model. By combining insights from exploration with scale and the new "Colossal Clean Crawled Corpus", the team achieved state-of-the-art results on many benchmarks covering summarization, question answering, and text classification, and released the dataset, pre-trained models, and code.
T5 came in a family of sizes. The original T5 work implemented five different model sizes — Small, Base, Large, 3B, and 11B — with the base and large models having 220 million and 770 million parameters respectively. The largest variant proved the concept dramatically: the largest model had 11 billion parameters and achieved state-of-the-art on the GLUE, SuperGLUE, SQuAD, and CNN/Daily Mail benchmarks, with a near-human score on SuperGLUE, a benchmark specifically designed to be difficult for machine learning models but easy for humans.
Where Word2Vec and static Word embedding techniques gave words fixed meanings, and BERT gave contextual meaning to spans, T5 pushed the frontier toward flexible generation, later inspiring specialized descendants like biomedical and multilingual text-to-text models built atop its framework.