Epstein Files Full PDF

A generative pre-trained transformer (GPT) is a type of large language model (LLM)^[1][2][3] that is widely used in generative artificial intelligence chatbots.^[4][5] GPTs are based on a deep learning architecture called the transformer. They are pre-trained on large datasets of unlabeled content, and able to generate novel content.^[2][3]

OpenAI was the first to apply generative pre-training to the transformer architecture, introducing the GPT-1 model in 2018.^[6] The company has since released many bigger GPT models. The chatbot ChatGPT, released in late 2022 (using GPT-3.5), was followed by many competitor chatbots using their own generative pre-trained transformers to generate text, such as Gemini, DeepSeek and Claude.

GPTs are primarily used to generate text, but can be trained to generate other kinds of data. For example, GPT-4o can process and generate text, images and audio.^[7] To improve performance on complex tasks, some GPTs, such as OpenAI o3, allocate more computation time analyzing the problem before generating an output, and are called reasoning models. In 2025, GPT-5 was released with a router that automatically selects whether to use a faster model or slower reasoning model based on the provided task.

During the 2010s, improved machine learning algorithms, more powerful computers, and an increase in the amount of digitized material allowed for an AI boom.^[8]

Separately, the concept of generative pre-training (GP) was a long-established technique in machine learning. GP is a form of self-supervised learning wherein a model is first trained on a large, unlabeled dataset (the "pre-training" step) to learn to generate data points. This pre-trained model is then adapted to a specific task using a labeled dataset (the "fine-tuning" step).^[9]

The transformer architecture for deep learning is the core technology of a GPT. Developed by researchers at Google, it was introduced in the paper "Attention Is All You Need", which was released on June 12, 2017. The transformer architecture solved many of the performance issues that were associated with older recurrent neural network (RNN) designs for natural language processing (NLP). The architecture's use of an attention mechanism allows models to process entire sequences of text at once, enabling the training of much larger and more sophisticated models. Since 2017, available transformer-based NLP systems have been capable of processing, mining, organizing, connecting, contrasting, and summarizing texts as well as answering questions from textual input.^{[citation needed]}

On June 11, 2018, OpenAI researchers and engineers published a paper called "Improving Language Understanding by Generative Pre-Training", which introduced GPT-1, the first GPT model.^[10] It was designed as a transformer-based large language model that used generative pre-training (GP) on BookCorpus, a diverse text corpus, followed by discriminative fine-tuning to focus on specific language tasks.^[11] This semi-supervised approach was seen as a breakthrough. Previously, the best-performing neural models in natural language processing (NLP) had commonly employed supervised learning from large amounts of manually labeled data – training a large language model with this approach would have been prohibitively expensive and time-consuming.^[10]

On February 14, 2019, OpenAI introduced GPT-2, a larger model that could generate coherent text. Created as a direct scale-up of its predecessor, it had both its parameter count and dataset size increased by a factor of 10. GPT-2 has 1.5 billion parameters and was trained on WebText, a 40-gigabyte dataset of 8 million web pages.^[12][13] Citing risks of malicious use, OpenAI opted for a "staged release", initially publishing smaller versions of the model before releasing the full 1.5-billion-parameter model in November.^[14]

On February 10, 2020, Microsoft introduced its Turing Natural Language Generation, which it claimed was the "largest language model ever published at 17 billion parameters." The model outperformed all previous language models at a variety of tasks, including summarizing texts and answering questions.^[15]

On May 28, 2020, OpenAI introduced GPT-3, a model with 175 billion parameters that was trained on a larger dataset compared to GPT-2. It marked a significant advancement in few-shot and zero-shot learning abilities. With few examples, it could perform various tasks that it was not explicitly trained for.^[16][17]

Following the release of GPT-3, OpenAI started using reinforcement learning from human feedback (RLHF) to align models' behavior more closely with human preferences. This led to the development of InstructGPT, a fine-tuned version of GPT-3. OpenAI further refined InstructGPT to create ChatGPT, the flagship chatbot product of OpenAI that was launched on November 30, 2022.^[18] ChatGPT was initially based on GPT-3.5, but it was later transitioned to the GPT-4 model, which was released on March 14, 2023.^[19][20] GPT-4 was also integrated into parts of several applications, including Microsoft Copilot, GitHub Copilot, Snapchat, Khan Academy, and Duolingo.^[21]

The immense popularity of ChatGPT spurred widespread development of competing GPT-based systems from other organizations. EleutherAI released a series of open-weight models, including GPT-J in 2021. Other major technology companies later developed their own GPT models, such as Google's PaLM and Gemini as well as Meta AI's Llama.^[22]

Many subsequent GPT models have been trained to be multimodal (able to process or to generate multiple types of data). For example, GPT-4o can both process and generate text, images, and audio.^[23] Additionally, GPT models like o3 and DeepSeek R1 have been trained with reinforcement learning to generate multi-step chain-of-thought reasoning before producing a final answer, which helps to solve complex problems in domains such as mathematics.^[24]

On August 7, 2025, OpenAI released GPT-5, which includes a router that automatically selects whether to use a faster model or slower reasoning model based on task.^[25][26]

A foundation model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks.^[27][28]

The most recent OpenAI's GPT-n series model is GPT-5.^[29]

Other such models include Google's PaLM, a broad foundation model that has been compared to GPT-3 and has been made available to developers via an API,^[30][31] and Together's GPT-JT, which has been reported as the closest-performing open-source alternative to GPT-3 (and is derived from earlier open-source GPTs).^[32] Meta AI (formerly Facebook) also has a generative transformer-based foundational large language model, known as LLaMA.^[33]

Foundational GPTs can also employ modalities other than text, for input and/or output. GPT-4 is a multi-modal LLM that is capable of processing text and image input (though its output is limited to text).^[34] Regarding multimodal output, some generative transformer-based models are used for text-to-image technologies such as diffusion^[35] and parallel decoding.^[36] Such kinds of models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.^[37]

A foundational GPT model can be further adapted to produce more targeted systems directed to specific tasks and/or subject-matter domains. Methods for such adaptation can include additional fine-tuning (beyond that done for the foundation model) as well as certain forms of prompt engineering.^[40]

An important example of this is fine-tuning models to follow instructions, which is of course a fairly broad task but more targeted than a foundation model. In January 2022, OpenAI introduced "InstructGPT" – a series of models which were fine-tuned to follow instructions using a combination of supervised training and reinforcement learning from human feedback (RLHF) on base GPT-3 language models.^[41][42] Advantages this had over the bare foundational models included higher accuracy, less negative/toxic sentiment, and generally better alignment with user needs. Hence, OpenAI began using this as the basis for its API service offerings.^[43] Other instruction-tuned models have been released by others, including a fully open version.^[44][45]

Another (related) kind of task-specific models are chatbots, which engage in human-like conversation. In November 2022, OpenAI launched ChatGPT – an online chat interface powered by an instruction-tuned language model trained in a similar fashion to InstructGPT.^[46] They trained this model using RLHF, with human AI trainers providing conversations in which they played both the user and the AI, and mixed this new dialogue dataset with the InstructGPT dataset for a conversational format suitable for a chatbot. Other major chatbots currently include Microsoft's Bing Chat, which uses OpenAI's GPT-4 (as part of a broader close collaboration between OpenAI and Microsoft),^[47] and Google's competing chatbot Gemini (initially based on their LaMDA family of conversation-trained language models, with plans to switch to PaLM).^[48]

Yet another kind of task that a GPT can be used for is the meta-task of generating its own instructions, like developing a series of prompts for 'itself' to be able to effectuate a more general goal given by a human user.^[49] This is known as an AI agent, and more specifically a recursive one because it uses results from its previous self-instructions to help it form its subsequent prompts; the first major example of this was Auto-GPT (which uses OpenAI's GPT models), and others have since been developed as well.^[50]

GPT systems can be directed toward particular fields or domains. Some reported examples of such models and apps are as follows:

• EinsteinGPT – for sales and marketing domains, to aid with customer relationship management (uses GPT-3.5)^[51][52]
• BloombergGPT – for the financial domain, to aid with financial news and information (uses "freely available" AI methods, combined with their proprietary data)^[53]
• Khanmigo – described as a GPT version for tutoring, in the education domain, it aids students using Khan Academy by guiding them through their studies without directly providing answers (powered by GPT-4)^[54][55]
• SlackGPT – for the Slack instant-messaging service, to aid with navigating and summarizing discussions on it (uses OpenAI's API)^[56]
• BioGPT – for the biomedical domain, to aid with biomedical literature text generation and mining (uses GPT-2)^[57]

Sometimes domain-specificity is accomplished via software plug-ins or add-ons. For example, several different companies have developed particular plugins that interact directly with OpenAI's ChatGPT interface,^[58][59] and Google Workspace has available add-ons such as "GPT for Sheets and Docs" – which is reported to aid use of spreadsheet functionality in Google Sheets.^[60][61]

OpenAI, which created the first generative pre-trained transformer (GPT) in 2018, asserted in 2023 that "GPT" should be regarded as a brand of OpenAI.^[62] In April 2023, OpenAI revised the brand guidelines in its terms of service to indicate that other businesses using its API to run their AI services would no longer be able to include "GPT" in such names or branding.^[63] In May 2023, OpenAI engaged a brand management service to notify its API customers of this policy, although these notifications stopped short of making overt legal claims (such as allegations of trademark infringement or demands to cease and desist).^[62] As of November 2023, OpenAI still prohibits its API licensees from naming their own products with "GPT",^[64] but it has begun enabling its ChatGPT Plus subscribers to make "custom versions of ChatGPT" called GPTs on the OpenAI site.^[65] OpenAI's terms of service says that its subscribers may use "GPT" in the names of these, although it's "discouraged".^[64]

Relatedly, OpenAI has applied to the United States Patent and Trademark Office (USPTO) to seek domestic trademark registration for the term "GPT" in the field of AI.^[62] OpenAI sought to expedite handling of its application, but the USPTO declined that request in April 2023.^[66] In May 2023, the USPTO responded to the application with a determination that "GPT" was both descriptive and generic.^[67] As of November 2023, OpenAI continues to pursue its argument through the available processes. Regardless, failure to obtain a registered U.S. trademark does not preclude some level of common-law trademark rights in the U.S.^[68] and trademark rights in other countries.^[69]

For any given type or scope of trademark protection in the U.S., OpenAI would need to establish that the term is actually "distinctive" to their specific offerings in addition to being a broader technical term for the kind of technology. Some media reports suggested in 2023 that OpenAI may be able to obtain trademark registration based indirectly on the fame of its GPT-based chatbot product, ChatGPT,^[66][70] for which OpenAI has separately sought protection (and which it has sought to enforce more strongly).^[71] Other reports have indicated that registration for the bare term "GPT" seems unlikely to be granted,^[62][72] as it is used frequently as a common term to refer simply to AI systems that involve generative pre-trained transformers.^{[3][73][74][75]} In any event, to whatever extent exclusive rights in the term may occur the U.S., others would need to avoid using it for similar products or services in ways likely to cause confusion.^[72][76] If such rights ever became broad enough to implicate other well-established uses in the field, the trademark doctrine of descriptive fair use could still continue non-brand-related usage.^[77]

In the European Union, the European Union Intellectual Property Office registered "GPT" as a trade mark of OpenAI in spring 2023. However, since spring 2024 the registration is being challenged and is pending cancellation.^[78]

In Switzerland, the Swiss Federal Institute of Intellectual Property registered "GPT" as a trade mark of OpenAI in spring 2023.^[79][80]

• Cyc
• Vision transformer