Generative pre-trained transformer

This article is about the general class of large language model. For OpenAI's line of GPT models, see GPT (OpenAI model).

Original GPT model

A generative pre-trained transformer (GPT) is a type of large language model (LLM) that is widely used in generative artificial intelligence chatbots. GPTs are based on a deep learning architecture called the transformer. They are pre-trained on large datasets of unlabeled content, and able to generate novel content.

OpenAI was the first to apply generative pre-training to the transformer architecture, introducing the GPT-1 model in 2018. The chatbot ChatGPT, released in late 2022 (using GPT-3.5), was followed by many competitor chatbots using their own generative pre-trained transformers to generate text, such as Gemini, DeepSeek and Claude.

GPTs are primarily used to generate text, but can be trained to generate other kinds of data. Multimodal GPTs can process or generate multiple types of data, such as text, images and audio. To improve performance on complex tasks, reasoning GPTs allocate more computation time analyzing the problem before generating an output.

Background

During the 2010s, improved machine learning algorithms, more powerful computers, and an increase in the amount of digitized material allowed for an AI boom.

Separately, the concept of generative pre-training (GP) was a long-established technique in machine learning. GP is a form of self-supervised learning wherein a model is first trained on a large, unlabeled dataset (the "pre-training" step) to learn to generate data points. This pre-trained model is then adapted to a specific task using a labeled dataset (the "fine-tuning" step).

The transformer architecture for deep learning is the core technology of a GPT. Developed by researchers at Google, it was introduced in the paper "Attention Is All You Need", which was released in 2017. The transformer architecture solved many of the performance issues that were associated with older recurrent neural network (RNN) designs for natural language processing (NLP). The architecture's use of an attention mechanism allows models to process entire sequences of text at once, enabling the training of much larger and more sophisticated models. Since 2017, available transformer-based NLP systems have been capable of processing, mining, organizing, connecting, contrasting, and summarizing texts as well as answering questions from textual input.

History

On June 11, 2018, OpenAI researchers and engineers published a paper called "Improving Language Understanding by Generative Pre-Training", which introduced GPT-1, the first GPT model. It was designed as a transformer-based large language model that used generative pre-training (GP) on BookCorpus, a diverse text corpus, followed by discriminative fine-tuning to focus on specific language tasks. This semi-supervised approach was seen as a breakthrough. GPT models use only the decoder part of the transformer architecture, pre-trained to predict the next token in a sequence. Previously, the best-performing neural models in natural language processing (NLP) had commonly employed supervised learning from large amounts of manually labeled data – training a large language model with this approach would have been prohibitively expensive and time-consuming.

On February 14, 2019, OpenAI introduced GPT-2, a larger model that could generate coherent text. Created as a direct scale-up of its predecessor, it had both its parameter count and dataset size increased by a factor of 10. GPT-2 has 1.5 billion parameters and was trained on WebText, a 40-gigabyte dataset of 8 million web pages. Citing risks of malicious use, OpenAI opted for a "staged release", initially publishing smaller versions of the model before releasing the full 1.5-billion-parameter model in November.

On February 10, 2020, Microsoft introduced its Turing Natural Language Generation, which it claimed was the "largest language model ever published at 17 billion parameters." The model outperformed all previous language models at a variety of tasks, including summarizing texts and answering questions.

On May 28, 2020, OpenAI introduced GPT-3, a model with 175 billion parameters that was trained on a larger dataset compared to GPT-2. It marked a significant advancement in few-shot and zero-shot learning abilities. With few examples, it could perform various tasks that it was not explicitly trained for.

Following the release of GPT-3, OpenAI started using reinforcement learning from human feedback (RLHF) to align models' behavior more closely with human preferences. This led to the development of InstructGPT, a fine-tuned version of GPT-3. OpenAI further refined InstructGPT to create ChatGPT, the flagship chatbot product of OpenAI that was launched on November 30, 2022. ChatGPT was initially based on GPT-3.5, but it was later transitioned to the GPT-4 model, which was released on March 14, 2023. GPT-4 was also integrated into parts of several applications, including Microsoft Copilot, GitHub Copilot, Snapchat, Khan Academy, and Duolingo.

The immense popularity of ChatGPT spurred widespread development of competing GPT-based systems from other organizations. EleutherAI released a series of open-weight models, including GPT-J in 2021. Other major technology companies later developed their own GPT models, such as Google's PaLM and Gemini as well as Meta AI's Llama.

Many subsequent GPT models have been trained to be multimodal (able to process or to generate multiple types of data). For example, GPT-4o can both process and generate text, images, and audio. Additionally, GPT models like o3 and DeepSeek R1 have been trained with reinforcement learning to generate multi-step chain-of-thought reasoning before producing a final answer, which helps to solve complex problems in domains such as mathematics.

Foundation models

A foundation model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks. Foundational GPTs can also employ modalities other than text, for input and/or output. Regarding multimodal output, some generative transformer-based models are used for text-to-image technologies such as diffusion and parallel decoding. Such kinds of models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.

Efficient transformer architectures

The computational and memory requirements of transformer-based models increase significantly as they scale to larger sizes and longer input sequences. The standard self-attention mechanism has a quadratic complexity with respect to input sequence length, as described in Attention Is All You Need.

Researchers proposed a number of efficiency improvements like sparse attention mechanisms and memory-efficient architectures that reduce computational costs while supporting longer context windows. Models like BigBird, Reformer, and FlashAttention demonstrate structured attention patterns or optimized computation to improve scalability and efficiency.

This has helped large language models to efficiently process long input sequences with reduced memory and computation during both training and inference.

Scaling laws

Scaling laws describe empirical relationships between the performance of large language models and factors such as model size, dataset size, and computational resources. Empirical work has found that performance tends to follow approximate power-law relationships as these factors increase.

Larger models trained on more data generally achieve lower training loss and better generalization. Later work suggests that performance is not determined by parameter count alone, but by how model size, data, and compute are balanced during training.

These observations have influenced the development of successive GPT models, particularly in decisions about architecture design, dataset composition, and training strategies.

Emergent abilities

Emergent abilities refer to capabilities that appear in large language models only when they reach a certain scale and are not present in smaller versions of the same models. These abilities are considered "emergent" because they arise as model size, training data, and compute increase.

Examples of emergent abilities include multi-step reasoning, in-context learning (the ability to perform tasks based on examples provided in prompts without additional training), and improved performance on complex language and reasoning benchmarks.

Research suggests that these capabilities do not scale linearly, but instead appear once models exceed certain thresholds in size and training scale.

This phenomenon has influenced the development of larger GPT models and contributed to their increased effectiveness across a wide range of tasks.

Brand issues

OpenAI claims "GPT" to be its own branding, citing its association with ChatGPT and its model versions' designations.

OpenAI, which created the first generative pre-trained transformer (GPT) in 2018, asserted in 2023 that "GPT" should be regarded as a brand of OpenAI. In April 2023, OpenAI revised the brand guidelines in its terms of service to indicate that other businesses using its API to run their AI services would no longer be able to include "GPT" in such names or branding. In May 2023, OpenAI engaged a brand management service to notify its API customers of this policy, although these notifications stopped short of making overt legal claims (such as allegations of trademark infringement or demands to cease and desist). As of November 2023, OpenAI still prohibits its API licensees from naming their own products with "GPT", but it has begun enabling its ChatGPT Plus subscribers to make "custom versions of ChatGPT" called GPTs on the OpenAI site. OpenAI's terms of service says that its subscribers may use "GPT" in the names of these, although it's "discouraged".

Relatedly, OpenAI has applied to the United States Patent and Trademark Office (USPTO) to seek domestic trademark registration for the term "GPT" in the field of AI. OpenAI sought to expedite handling of its application, but the USPTO declined that request in April 2023. In May 2023, the USPTO rejected OpenAI's trademark application, determining that "GPT" was both descriptive and generic.

In the European Union, the European Union Intellectual Property Office registered "GPT" as a trade mark of OpenAI in spring 2023. However, since spring 2024 the registration is being challenged and is pending cancellation. In Switzerland, the Swiss Federal Institute of Intellectual Property registered "GPT" as a trade mark of OpenAI in spring 2023. The China National Intellectual Property Administration denied OpenAI's applications to trademark the terms "GPT" and "ChatGPT" in 2024, and the term "GPT-5" in August 2025, after which OpenAI filed appeals.

Evaluation and benchmarking

The evaluation of generative pre-trained transformer models is carried out using a range of benchmarks and metrics, which aim to assess their performance across different tasks. Common approaches include accuracy on standard datasets, as well as other features such as robustness, bias, and toxicity.

These models are typically tested on tasks such as natural language understanding, reasoning, answering queries, and code generation. Sometimes they combine multiple tasks to provide a broader assessment of model performance across domains.

More recent approaches extend these evaluations to include other features such as fairness, efficiency, and transparency, in order to get a more accurate assessment of these models.

Evaluation remains an active area of research, as existing tests may not accurately reflect real-world performance or the risks associated with large-scale generative models.

Ethical considerations and societal impact

Generative pre-trained transformer models have raised a range of ethical and societal concerns, particularly regarding bias, misinformation, and environmental impact. Large language models can reproduce and amplify patterns present in their training data, including social biases, which may lead to unfair or misleading outputs.

These models have also been associated with generating inaccurate or misleading information as they are designed to produce fluent text rather than to verify factual accuracy. This has an impact on their use in applications such as automated content generation and information dissemination.

Training large-scale models also requires huge computational resources, contributing to increased energy consumption and environmental costs. Concerns about the environmental impact of large AI systems have led to calls for more efficient training methods and more transparency in reporting resource usage.

See also

• Vision transformer