GPT-4 Technical Report (2024)

1 Introduction

This technical report presents GPT-4, a large multimodal model capable of processing image and text inputs and producing text outputs. Such models are an important area of study as they have the potential to be used in a wide range of applications, such as dialogue systems, text summarization, and machine translation. As such, they have been the subject of substantial interest and progress in recent years[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34].

One of the main goals of developing such models is to improve their ability to understand and generate natural language text, particularly in more complex and nuanced scenarios.To test its capabilities in such scenarios, GPT-4 was evaluated on a variety of exams originally designed for humans. In these evaluations it performs quite well and often outscores the vast majority of human test takers. For example, on a simulated bar exam, GPT-4 achieves a score that falls in the top 10% of test takers. This contrasts with GPT-3.5, which scores in the bottom 10%.

On a suite of traditional NLP benchmarks, GPT-4 outperforms both previous large language models and most state-of-the-art systems (which often have benchmark-specific training or hand-engineering). On the MMLU benchmark[35, 36], an English-language suite of multiple-choice questions covering 57 subjects, GPT-4 not only outperforms existing models by a considerable margin in English, but also demonstrates strong performance in other languages. On translated variants of MMLU, GPT-4 surpasses the English-language state-of-the-art in 24 of 26 languages considered. We discuss these model capability results, as well as model safety improvements and results, in more detail in later sections.

This report also discusses a key challenge of the project, developing deep learning infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to make predictions about the expected performance of GPT-4 (based on small runs trained in similar ways) that were tested against the final run to increase confidence in our training.

Despite its capabilities, GPT-4 has similar limitations to earlier GPT models[1, 37, 38]: it is not fully reliable (e.g. can suffer from “hallucinations”), has a limited context window, and does not learn from experience. Care should be taken when using the outputs of GPT-4, particularly in contexts where reliability is important.

GPT-4’s capabilities and limitations create significant and novel safety challenges, and we believe careful study of these challenges is an important area of research given the potential societal impact. This report includes an extensive system card (after the Appendix) describing some of the risks we foresee around bias, disinformation, over-reliance, privacy, cybersecurity, proliferation, and more. It also describes interventions we made to mitigate potential harms from the deployment of GPT-4, including adversarial testing with domain experts, and a model-assisted safety pipeline.

2 Scope and Limitations of this Technical Report

This report focuses on the capabilities, limitations, and safety properties of GPT-4. GPT-4 is a Transformer-style model[39] pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers. The model was then fine-tuned using Reinforcement Learning from Human Feedback (RLHF)[40]. Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

We are committed to independent auditing of our technologies, and shared some initial steps and ideas in this area in the system card accompanying this release.¹¹1In addition to the accompanying system card, OpenAI will soon publish additional thoughts on the social and economic implications of AI systems, including the need for effective regulation. We plan to make further technical details available to additional third parties who can advise us on how to weigh the competitive and safety considerations above against the scientific value of further transparency.

3 Predictable Scaling

A large focus of the GPT-4 project was building a deep learning stack that scales predictably. The primary reason is that for very large training runs like GPT-4, it is not feasible to do extensive model-specific tuning. To address this, we developed infrastructure and optimization methods that have very predictable behavior across multiple scales. These improvements allowed us to reliably predict some aspects of the performance of GPT-4 from smaller models trained using $1,000\times$ – $10,000\times$ less compute.

3.1 Loss Prediction

The final loss of properly-trained large language models is thought to be well approximated by power laws in the amount of compute used to train the model[41, 42, 2, 14, 15].

To verify the scalability of our optimization infrastructure, we predicted GPT-4’s final loss on our internal codebase (not part of the training set) by fitting a scaling law with an irreducible loss term (as inHenighan etal. [15]): $L(C)=aC^{b}+c,$ from models trained using the same methodology but using at most 10,000x less compute than GPT-4. This prediction was made shortly after the run started, without use of any partial results. The fitted scaling law predicted GPT-4’s final loss with high accuracy (Figure 1).

3.2 Scaling of Capabilities on HumanEval

Having a sense of the capabilities of a model before training can improve decisions around alignment, safety, and deployment. In addition to predicting final loss, we developed methodology to predict more interpretable metrics of capability. One such metric is pass rate on the HumanEval dataset[43], which measures the ability to synthesize Python functions of varying complexity. We successfully predicted the pass rate on a subset of the HumanEval dataset by extrapolating from models trained with at most $1,000\times$ less compute (Figure 2).

For an individual problem in HumanEval, performance may occasionally worsen with scale. Despite these challenges, we find an approximate power law relationship $-\mathrm{E}_{P}[\log(\mathrm{pass\_rate(C)})]=\alpha*\mathrm{C}^{-k}$ where $k$ and $\alpha$ are positive constants, and $P$ is a subset of problems in the dataset. We hypothesize that this relationship holds for all problems in this dataset. In practice, very low pass rates are difficult or impossible to estimate, so we restrict to problems $P$ and models $M$ such that given some large sample budget, every problem is solved at least once by every model.

We registered predictions for GPT-4’s performance on HumanEval before training completed, using only information available prior to training. All but the 15 hardest HumanEval problems were split into 6 difficulty buckets based on the performance of smaller models. The results on the $3^{\mathrm{rd}}$ easiest bucket are shown in Figure 2, showing that the resulting predictions were very accurate for this subset of HumanEval problems where we can accurately estimate $\log(\mathrm{pass\_rate})$ for several smaller models. Predictions on the other five buckets performed almost as well, the main exception being GPT-4 underperforming our predictions on the easiest bucket.

Certain capabilities remain hard to predict. For example, the InverseScaling Prize[44] proposed several tasks for which model performance decreases as a function of scale. Similarly to a recent result byWei etal. [45], we find that GPT-4 reverses this trend, as shown on one of the tasks called Hindsight Neglect[46] in Figure 3.

We believe that accurately predicting future capabilities is important for safety. Going forward we plan to refine these methods and register performance predictions across various capabilities before large model training begins, and we hope this becomes a common goal in the field.

4 Capabilities

We tested GPT-4 on a diverse set of benchmarks, including simulating exams that were originally designed for humans.³³3We used the post-trained RLHF model for these exams. We did no specific training for these exams. A minority of the problems in the exams were seen by the model during training; for each exam we run a variant with these questions removed and report the lower score of the two. We believe the results to be representative. For further details on contamination (methodology and per-exam statistics), see Appendix C.

Exams were sourced from publicly-available materials. Exam questions included both multiple-choice and free-response questions; we designed separate prompts for each format, and images were included in the input for questions which required it. The evaluation setup was designed based on performance on a validation set of exams, and we report final results on held-out test exams. Overall scores were determined by combining multiple-choice and free-response question scores using publicly available methodologies for each exam. We estimate and report the percentile each overall score corresponds to.See AppendixA for further details on the exam evaluation methodology.

GPT-4 exhibits human-level performance on the majority of these professional and academic exams. Notably, it passes a simulated version of the Uniform Bar Examination with a score in the top 10% of test takers (Table1, Figure4).

The model’s capabilities on exams appear to stem primarily from the pre-training process and are not significantly affected by RLHF. On multiple choice questions, both the base GPT-4 model and the RLHF model perform equally well on average across the exams we tested (see AppendixB).

We also evaluated the pre-trained base GPT-4 model on traditional benchmarks designed for evaluating language models. For each benchmark we report, we ran contamination checks for test data appearing in the training set (see AppendixD for full details on per-benchmark contamination).⁴⁴4During our contamination check we discovered that portions of BIG-bench[48] were inadvertently mixed into the training set, and we excluded it from our reported results. We used few-shot prompting [1] for all benchmarks when evaluating GPT-4.⁵⁵5For GSM-8K, we include part of the training set in GPT-4’s pre-training mix (see AppendixE for details). We use chain-of-thought prompting[11] when evaluating.

GPT-4 considerably outperforms existing language models, as well as previously state-of-the-art (SOTA) systems whichoften have benchmark-specific crafting or additional training protocols (Table2).

Many existing ML benchmarks are written in English. To gain an initial understanding of GPT-4’s capabilities in other languages, we translated the MMLU benchmark[35, 36] – a suite of multiple-choice problems spanning 57 subjects – into a variety of languages using Azure Translate (see AppendixF for example translations and prompts). We find that GPT-4 outperforms the English-language performance of GPT 3.5 andexisting language models (Chinchilla[2] and PaLM[3]) for the majority of languages wetested, including low-resource languages such as Latvian, Welsh, and Swahili (Figure5).

GPT-4 substantially improves over previous models in the ability to follow user intent[63]. On a dataset of 5,214 prompts submitted to ChatGPT[64] and the OpenAI API[47], the responses generated by GPT-4 were preferred over the responses generated by GPT-3.5 on $70.2\%$ of prompts.⁶⁶6We collected user prompts sent to us through ChatGPT and the OpenAI API, sampled one response from each model, and sent these prompts and responses to human labelers. The labelers were instructed to judge whether the response is what the user would have wanted given the prompt. The labelers were not told which response was generated by which model and the order in which the responses were presented was randomised. We filter out prompts containing any kind of disallowed or sensitive content, including personally identifiable information (PII), sexual content, hate-speech, and similar content. We also filter short (e.g. ”Hello, ChatGPT!”) and overly-common prompts.

We are open-sourcing OpenAI Evals⁷⁷7https://github.com/openai/evals, our framework for creating and running benchmarks for evaluating models like GPT-4 while inspecting performance sample by sample. Evals is compatible with existing benchmarks, and can be used to track performance of models in deployment. We plan to increase the diversity of these benchmarks over time to represent a wider set of failure modes and a harder set of tasks.

5 Limitations

Despite its capabilities, GPT-4 has similar limitations as earlier GPT models. Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors). Great care should be taken when using language model outputs, particularly in high-stakes contexts, with the exact protocol (such as human review, grounding with additional context, or avoiding high-stakes uses altogether) matching the needs of specific applications. See our System Card for details.

GPT-4 significantly reduces hallucinations relative to previous GPT-3.5 models (which have themselves been improving with continued iteration). GPT-4 scores 19 percentage points higher than our latest GPT-3.5 on our internal, adversarially-designed factuality evaluations (Figure6).

GPT-4 makes progress on public benchmarks like TruthfulQA[66], which tests the model’s ability to separate fact from an adversarially-selected set of incorrect statements (Figure7). These questions are paired with factually incorrect answers that are statistically appealing. The GPT-4 base model is only slightly better at this task than GPT-3.5; however, after RLHF post-training we observe large improvements over GPT-3.5.⁸⁸8We did not check the RLHF post-training data for contamination with TruthfulQA Table4 shows both a correct and an incorrect answer. GPT-4 resists selecting common sayings (you can’t teach an old dog new tricks), however it still can miss subtle details (Elvis Presley was not the son of an actor, so Perkins is the correct answer).

GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its pre-training data cuts off in September 2021⁹⁹9The pre-training and post-training data contain a small amount of more recent data, and does not learn from its experience. It can sometimes make simple reasoning errors which do not seem to comport with competence across so many domains, or be overly gullible in accepting obviously false statements from a user. It can fail at hard problems the same way humans do, such as introducing security vulnerabilities into code it produces.

GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced (Figure8).

GPT-4 has various biases in its outputs that we have taken efforts to correct but which will take some time to fully characterize and manage.We aim to make GPT-4 and other systems we build have reasonable default behaviors that reflect a wide swath of users’ values, allow those systems to be customized within some broad bounds, and get public input on what those bounds should be. SeeOpenAI [68] for more details.

6 Risks & mitigations

We invested significant effort towards improving the safety and alignment of GPT-4. Here we highlight our use of domain experts for adversarial testing and red-teaming, and our model-assisted safety pipeline [69]and the improvement in safety metrics over prior models.

Adversarial Testing via Domain Experts: GPT-4 poses similar risks as smaller language models, such as generating harmful advice, buggy code, or inaccurate information. However, the additional capabilities of GPT-4 lead to new risk surfaces. To understand the extent of these risks, we engaged over 50 experts from domains such as long-term AI alignment risks, cybersecurity, biorisk, and international security to adversarially test the model. Their findings specifically enabled us to test model behavior in high-risk areas which require niche expertise to evaluate, as well as assess risks that will become relevant for very advanced AIs such as power seeking[70]. Recommendations and training data gathered from these experts fed into our mitigations and improvements for the model; for example, we’ve collected additional data to improve GPT-4’s ability to refuse requests on how to synthesize dangerous chemicals (Table 5).

Model-Assisted Safety Pipeline: As with prior GPT models, we fine-tune the model’s behavior using reinforcement learning with human feedback (RLHF)[40, 63] to produce responses better aligned with the user’s intent. However, after RLHF, our models can still be brittle on unsafe inputs as well as sometimes exhibit undesired behaviors on both safe and unsafe inputs. These undesired behaviors can arise when instructions to labelers were underspecified during reward model data collection portion of the RLHF pipeline. When given unsafe inputs, the model may generate undesirable content, such as giving advice on committing crimes. Furthermore, the model may also become overly cautious on safe inputs, refusing innocuous requests or excessively hedging. To steer our models towards appropriate behaviour at a more fine-grained level, we rely heavily on our models themselves as tools. Our approach to safety consists of two main components, an additional set of safety-relevant RLHF training prompts, and rule-based reward models (RBRMs).

Our rule-based reward models (RBRMs) are a set of zero-shot GPT-4 classifiers. These classifiers provide an additional reward signal to the GPT-4 policy model during RLHF fine-tuning that targets correct behavior, such as refusing to generate harmful content or not refusing innocuous requests. The RBRM takes three inputs: the prompt (optional), the output from the policy model, and a human-written rubric (e.g., a set of rules in multiple-choice style) for how this output should be evaluated. Then, the RBRM classifies the output based on the rubric. For example, we can provide a rubric that instructs the model to classify a response as one of: (a) a refusal in the desired style, (b) a refusal in the undesired style (e.g., evasive or rambling), (c) containing disallowed content, or (d) a safe non-refusal response. Then on the set of safety-relevant training prompts, which request harmful content such as illicit advice, we can reward GPT-4 for refusing these requests. Conversely, we can reward GPT-4 for not refusing requests on a subset of prompts guaranteed to be safe and answerable. This technique is related to work byGlaese etal. [71] andPerez etal. [72]. This, combined with other improvements such as computing optimal RBRM weights and providing additional SFT data targeting the areas we want to improve, allowed us to steer the model closer towards the desired behaviour.

Improvements on Safety Metrics: Our mitigations have significantly improved many of GPT-4’s safety properties. We’ve decreased the model’s tendency to respond to requests for disallowed content (Table6) by 82% compared to GPT-3.5, and GPT-4 responds to sensitive requests (e.g., medical advice and self-harm, Table 7) in accordance with our policies 29% more often (Figure9). On the RealToxicityPrompts dataset[73], GPT-4 produces toxic generations only 0.73% of the time, while GPT-3.5 generates toxic content 6.48% of time.

Overall, our model-level interventions increase the difficulty of eliciting bad behavior but doing so is still possible. For example, there still exist “jailbreaks” (e.g., adversarial system messages, see Figure 10 in the System Card for more details) to generate content which violate our usage guidelines. So long as these limitations exist, it’s important to complement them with deployment-time safety techniques like monitoring for abuse as well as a pipeline for fast iterative model improvement.

GPT-4 and successor models have the potential to significantly influence society in both beneficial and harmful ways. We are collaborating with external researchers to improve how we understand and assess potential impacts, as well as to build evaluations for dangerous capabilities that may emerge in future systems. We will soon publish recommendations on steps society can take to prepare for AI’s effects and initial ideas for projecting AI’s possible economic impacts.

7 Conclusion

We characterize GPT-4, a large multimodal model with human-level performance on certain difficult professional and academic benchmarks. GPT-4 outperforms existing large language models on a collection of NLP tasks, and exceeds the vast majority of reported state-of-the-art systems (which often include task-specific fine-tuning). We find that improved capabilities, whilst usually measured in English, can be demonstrated in many different languages. We highlight how predictable scaling allowed us to make accurate predictions on the loss and capabilities of GPT-4.

GPT-4 presents new risks due to increased capability, and we discuss some of the methods and results taken to understand and improve its safety and alignment. Though there remains much work to be done, GPT-4 represents a significant step towards broadly useful and safely deployed AI systems.

Authorship, Credit Attribution, and Acknowledgements

Please cite this work as “OpenAI (2023)”.

Pretraining

Core contributors¹⁰¹⁰10All author lists sorted alphabetically.
Christopher BernerSupercomputing lead
Greg BrockmanInfrastructure lead
Trevor CaiThroughput lead
David FarhiManager of optimization team
Chris HesseInfrastructure usability co-lead
Shantanu JainInfrastructure usability co-lead
Kyle KosicUptime and stability lead
Jakub PachockiOverall lead, optimization lead
Alex PainoArchitecture & data vice lead
Mikhail PavlovSoftware correctness lead
Michael PetrovHardware correctness lead
Nick RyderArchitecture & data lead
Szymon SidorOptimization vice lead
Nikolas TezakExecution lead
Phil TilletTriton lead
Amin TootoonchianModel distribution, systems & networking lead
Qiming YuanDataset sourcing and processing lead
Wojciech ZarembaManager of dataset teamCompute cluster scaling¹⁰¹⁰10All author lists sorted alphabetically.
Christopher Berner, Oleg Boiko, Andrew Cann, Ben Chess, Christian Gibson, Mateusz Litwin, Emy Parparita, Henri Roussez, Eric Sigler, Akila WelihindaData¹⁰¹⁰10All author lists sorted alphabetically.
Sandhini Agarwal, Suchir Balaji, Mo Bavarian, Che Chang, Sheila Dunning, Leo Gao, Jonathan Gordon, Peter Hoeschele, Shawn Jain, Shantanu Jain, Roger Jiang, Heewoo Jun, Łukasz Kaiser, Nitish Shirish Keskar, Jong Wook Kim, Aris Konstantinidis, Chak Ming Li, Todor Markov, Bianca Martin, David Mély, Oleg Murk, Hyeonwoo Noh, Long Ouyang, Alex Paino, Vitchyr Pong, Alec Radford, Nick Ryder, John Schulman, Daniel Selsam, Ian Sohl, Chelsea Voss, Lilian Weng, Clemens Winter, Tao Xu, Qiming Yuan, Wojciech ZarembaDistributed training infrastructure¹⁰¹⁰10All author lists sorted alphabetically.
Greg Brockman, Trevor Cai, Chris Hesse, Shantanu Jain, Yongjik Kim, Kyle Kosic, Mateusz Litwin, Jakub Pachocki, Mikhail Pavlov, Szymon Sidor, Nikolas Tezak, Madeleine Thompson, Amin Tootoonchian, Qiming YuanHardware correctness¹⁰¹⁰10All author lists sorted alphabetically.
Greg Brockman, Shantanu Jain, Kyle Kosic, Michael Petrov, Nikolas Tezak, Amin Tootoonchian, Chelsea Voss, Qiming YuanOptimization & architecture¹⁰¹⁰10All author lists sorted alphabetically.
Igor Babuschkin, Mo Bavarian, Adrien Ecoffet, David Farhi, Jesse Han, Ingmar Kanitscheider, Daniel Levy, Jakub Pachocki, Alex Paino, Mikhail Pavlov, Nick Ryder, Szymon Sidor, Jie Tang, Jerry Tworek, Tao XuTraining run babysitting¹⁰¹⁰10All author lists sorted alphabetically.
Suchir Balaji, Mo Bavarian, Greg Brockman, Trevor Cai, Chris Hesse, Shantanu Jain, Roger Jiang, Yongjik Kim, Kyle Kosic, Mateusz Litwin, Jakub Pachocki, Alex Paino, Mikhail Pavlov, Michael Petrov, Nick Ryder, Szymon Sidor, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Chelsea Voss, Ben Wang, Tao Xu, Qiming Yuan

Long context

References

• Brown etal. [2020]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, JaredD. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, etal.Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
• Hoffmann etal. [2022]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego deLas Casas, LisaAnne Hendricks, Johannes Welbl, Aidan Clark, etal.Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022.
• Chowdhery etal. [2022]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, HyungWon Chung, Charles Sutton, Sebastian Gehrmann, etal.PaLM: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311, 2022.
• Rae etal. [2021]JackW Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, etal.Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446, 2021.
• Dai etal. [2019]Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, QuocV. Le, and Ruslan Salakhutdinov.Transformer-XL: Attentive language models beyond a fixed-length context.arXiv preprint arXiv:1901.02860, 2019.
• Liu etal. [2019]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019.
• Devlin etal. [2018]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.BERT: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018.
• Raffel etal. [2019]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ Liu.Exploring the limits of transfer learning with a unified text-to-text transformer.arXiv preprint arXiv:1910.10683, 2019.
• Shazeer and Stern [2018]Noam Shazeer and Mitchell Stern.Adafactor: Adaptive learning rates with sublinear memory cost.arXiv preprint arXiv:1804.04235, 2018.
• Ba etal. [2016]JimmyLei Ba, JamieRyan Kiros, and GeoffreyE. Hinton.Layer normalization.arXiv preprint arXiv:1607.06450, 2016.
• Wei etal. [2022a]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, EdChi, Quoc Le, and Denny Zhou.Chain-of-thought prompting elicits reasoning in large language models.NeurIPS, 2022a.
• Huang etal. [2022]Jiaxin Huang, ShixiangShane Gu, LeHou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han.Large language models can self-improve.arXiv preprint arXiv:2210.11610, 2022.
• Kojima etal. [2022]Takeshi Kojima, ShixiangShane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.Large language models are zero-shot reasoners.arXiv preprint arXiv:2205.11916, 2022.
• Kaplan etal. [2020]Jared Kaplan, Sam McCandlish, Tom Henighan, TomB. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.
• Henighan etal. [2020]Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, TomB. Brown, Prafulla Dhariwal, Scott Gray, etal.Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701, 2020.
• Yang etal. [2022]Greg Yang, EdwardJ. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao.Tensor Programs V: Tuning large neural networks via zero-shot hyperparameter transfer.arXiv preprint arXiv:2203.03466, 2022.
• Shazeer etal. [2017]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.Outrageously large neural networks: The sparsely-gated Mixture-of-Experts layer.arXiv preprint arXiv:1701.06538, 2017.
• Zoph etal. [2022]Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus.ST-MoE: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906, 2022.
• Wei etal. [2022b]Jason Wei, YiTay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, etal.Emergent abilities of large language models.TMLR, 2022b.
• Dehghani etal. [2019]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser.Universal transformers.In International Conference on Learning Representations, 2019.URL https://openreview.net/forum?id=HyzdRiR9Y7.
• Su etal. [2021]Jianlin Su, YuLu, Shengfeng Pan, Ahmed Murtadha, BoWen, and Yunfeng Liu.RoFormer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021.
• [22]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, etal.Flamingo: a visual language model for few-shot learning.In Advances in Neural Information Processing Systems.
• Chen etal. [2022a]XiChen, Xiao Wang, Soravit Changpinyo, AJPiergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, etal.PaLI: A jointly-scaled multilingual language-image model.arXiv preprint arXiv:2209.06794, 2022a.
• Wang and Komatsuzaki [2021]Ben Wang and Aran Komatsuzaki.GPT-J-6B: A 6 billion parameter autoregressive language model, 2021.
• Black etal. [2021]Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman.GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow.If you use this software, please cite it using these metadata, 58, 2021.
• Scao etal. [2022]TevenLe Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, AlexandraSasha Luccioni, François Yvon, Matthias Gallé, etal.Bloom: A 176B-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022.
• Zhang etal. [2022]Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, XiVictoria Lin, etal.OPT: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.
• Touvron etal. [2023]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, etal.LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023.
• Radford etal. [2017]Alec Radford, Rafal Józefowicz, and Ilya Sutskever.Learning to generate reviews and discovering sentiment.arXiv preprint arXiv:1704.01444, 2017.
• Lample and Conneau [2019]Guillaume Lample and Alexis Conneau.Cross-lingual language model pretraining.arXiv preprint arXiv:1901.07291, 2019.
• Dao etal. [2022]Tri Dao, DanielY. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.Flashattention: Fast and memory-efficient exact attention with io-awareness.arXiv preprint arXiv:2205.14135, 2022.
• Child etal. [2019]Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019.
• Rabe and Staats [2021]MarkusN. Rabe and Charles Staats.Self-attention does not need $o(n^{2})$ memory.arXiv preprint arXiv:2112.05682, 2021.
• Gray etal. [2017]Scott Gray, Alec Radford, and DiederikP. Kingma.Gpu kernels for block-sparse weights, 2017.URL https://cdn.openai.com/blocksparse/blocksparsepaper.pdf.
• Hendrycks etal. [2021a]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
• Hendrycks etal. [2021b]Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.Aligning AI with shared human values.Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
• Radford etal. [2019]Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.Language models are unsupervised multitask learners.2019.
• Radford etal. [2018]Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.Improving language understanding by generative pre-training.2018.
• Vaswani etal. [2017]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidanN Gomez, Łukasz Kaiser, and Illia Polosukhin.Attention is all you need.NeurIPS, 2017.
• Christiano etal. [2017]PaulF Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems, 30, 2017.
• Hestness etal. [2017]Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, MdPatwary, Mostofa Ali, Yang Yang, and Yanqi Zhou.Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409, 2017.
• Thompson etal. [2020]NeilC Thompson, Kristjan Greenewald, Keeheon Lee, and GabrielF Manso.The computational limits of deep learning.arXiv preprint arXiv:2007.05558, 2020.
• Chen etal. [2021]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, HenriquePonde deOliveiraPinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, FelipePetroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, WilliamHebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, AndrewN. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.Evaluating large language models trained on code.2021.
• McKenzie etal. [2022a]Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez.The Inverse Scaling Prize, 2022a.URL https://github.com/inverse-scaling/prize.
• Wei etal. [2022c]Jason Wei, Najoung Kim, YiTay, and QuocV. Le.Inverse scaling can become U-shaped.arXiv preprint arXiv:2211.02011, 2022c.
• McKenzie etal. [2022b]Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez.Inverse Scaling Prize: First round winners, 2022b.URL https://irmckenzie.co.uk/round1.
• Brockman etal. [2020]Greg Brockman, Peter Welinder, Mira Murati, and OpenAI.OpenAI: OpenAI API, 2020.URL https://openai.com/blog/openai-api.
• Srivastava etal. [2022]Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu AwalMd Shoeb, Abubakar Abid, Adam Fisch, AdamR. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, etal.Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615, 2022.
• Hendrycks etal. [2020]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020.
• Tay etal. [2022]YiTay, Jason Wei, HyungWon Chung, VinhQ Tran, DavidR So, Siamak Shakeri, Xavier Garcia, HuaixiuSteven Zheng, Jinfeng Rao, Aakanksha Chowdhery, etal.Transcending scaling laws with 0.1% extra compute.arXiv preprint arXiv:2210.11399, 2022.
• Chung etal. [2022]HyungWon Chung, LeHou, Shayne Longpre, Barret Zoph, YiTay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, etal.Scaling instruction-finetuned language models.arXiv preprint arXiv:2210.11416, 2022.
• Zellers etal. [2019]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.HellaSwag: Can a machine really finish your sentence?In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.doi: 10.18653/v1/P19-1472.URL https://aclanthology.org/P19-1472.
• Liu etal. [2020]Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, YuWang, Hoifung Poon, and Jianfeng Gao.Adversarial training for large neural language models.arXiv preprint arXiv:2004.08994, 2020.
• Clark etal. [2018]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.Think you have solved question answering? Try ARC, the AI2 reasoning challenge.ArXiv, abs/1803.05457, 2018.
• Wang etal. [2022]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, EdChi, and Denny Zhou.Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022.
• Sakaguchi etal. [2019]Keisuke Sakaguchi, RonanLe Bras, Chandra Bhagavatula, and Yejin Choi.WinoGrande: An adversarial Winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019.
• Chen etal. [2022b]Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen.CodeT: Code generation with generated tests.arXiv preprint arXiv:2207.10397, 2022b.
• Dua etal. [2019]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.doi: 10.18653/v1/N19-1246.URL https://aclanthology.org/N19-1246.
• Chen etal. [2020]Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, LeSong, Taifeng Wang, Yuan Qi, and Wei Chu.Question directed graph attention network for numerical reasoning over text.arXiv preprint arXiv:2009.07448, 2020.
• Cobbe etal. [2021]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021.
• Lewkowycz etal. [2022]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, etal.Solving quantitative reasoning problems with language models.arXiv preprint arXiv:2206.14858, 2022.
• Uesato etal. [2022]Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.Solving math word problems with process- and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022.
• Ouyang etal. [2022]Long Ouyang, Jeff Wu, XuJiang, Diogo Almeida, CarrollL Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, etal.Training language models to follow instructions with human feedback.arXiv preprint arXiv:2203.02155, 2022.
• OpenAI [2022]OpenAI.OpenAI: Introducing ChatGPT, 2022.URL https://openai.com/blog/chatgpt.
• OpenAI [2023a]OpenAI.OpenAI: GPT-4, 2023a.URL https://openai.com/research/gpt-4.
• Lin etal. [2022]Stephanie Lin, Jacob Hilton, and Owain Evans.TruthfulQA: Measuring how models mimic human falsehoods.In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguistics.doi: 10.18653/v1/2022.acl-long.229.URL https://aclanthology.org/2022.acl-long.229.
• Bai etal. [2022]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, etal.Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022.
• OpenAI [2023b]OpenAI.OpenAI: How should AI systems behave, and who should decide?, 2023b.URL https://openai.com/blog/how-should-ai-systems-behave.
• Leike etal. [2022]Jan Leike, John Schulman, and Jeffrey Wu.OpenAI: Our approach to alignment research, 2022.URL https://openai.com/blog/our-approach-to-alignment-research.
• Carlsmith [2022]Joseph Carlsmith.Is power-seeking AI an existential risk?ArXiv, abs/2206.13353, 2022.
• Glaese etal. [2022]Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, JaumeSanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, LisaAnne Hendricks, and Geoffrey Irving.Improving alignment of dialogue agents via targeted human judgements.arXiv preprint arXiv:2209.14375, 2022.
• Perez etal. [2022]Ethan Perez, Saffron Huang, H.Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022.
• Gehman etal. [2020]Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and NoahA Smith.RealToxicityPrompts: Evaluating neural toxic degeneration in language models.arXiv preprint arXiv:2009.11462, 2020.
• Seigel [2020]Dora Seigel.How do you calculate SAT score? raw and scaled, 1 2020.URL https://blog.prepscholar.com/how-to-calculate-sat-score.
• [75]The Albert blog.URL https://www.albert.io/blog/.
• ofAmerica [2023]MathematicalAssociation ofAmerica.AMC statistics, 2023.URL http://amc-reg.maa.org/Reports/GeneralReports.aspx.
• Edwards [2022]Halle Edwards.SAT percentiles and score rankings, 2022.URL https://blog.prepscholar.com/sat-percentiles-and-score-rankings.
• Board [2022a]College Board.Understanding SAT scores, 2022a.URL https://satsuite.collegeboard.org/media/pdf/understanding-sat-scores.pdf.
• Board [2022b]College Board.AP score distributions by subject, 2022b.URL https://apcentral.collegeboard.org/media/pdf/ap-score-distributions-by-subject-2022.pdf.
• for Excellencein Education [2022]Center for Excellencein Education.2020 USABO Semifinal exam score distribution, 2022.URL https://www.usabo-trc.org/sites/default/files/allfiles/2020%20USABO%20Semifinal%20Exam%20Histogram.pdf.
• Swimmer [2021]Chris Swimmer.GRE score percentiles – what does your score mean for you? (2021 update), 4 2021.URL https://magoosh.com/gre/gre-score-percentiles/.
• Nici [2020]JohnB. Nici.AP Art History: 5 Practice Tests + Comprehensive Review + Online Practice.Barron’s Test Prep. Barron’s Educational Series, 2020.ISBN 9781506260501.
• ETS [2022]ETS.GRE sample issue task, 2022.URL https://www.ets.org/pdfs/gre/sample-issue-task.pdf.
• Mitchell etal. [2019]Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, InioluwaDeborah Raji, and Timnit Gebru.Model Cards for Model Reporting.In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 220–229, January 2019.doi: 10.1145/3287560.3287596.
• Green etal. [2022]Nekesha Green, Chavez Procope, Adeel Cheema, and Adekunle Adediji.System Cards, a new resource for understanding how AI systems work.https://ai.facebook.com/blog/system-cards-a-new-resource-for-understanding-how-ai-systems-work/, February 2022.

Appendix A Exam Benchmark Methodology

Appendix B Impact of RLHF on capability

To test the impact of RLHF on the capability of our base model, we ran the multiple-choice question portions of our exam benchmark on the GPT-4 base model and the post RLHF GPT-4 model. The results are shown in Table8. Averaged across all exams, the base model achieves a score of 73.7% while the RLHF model achieves a score of 74.0%, suggesting that post-training does not substantially alter base model capability.

For free-response questions, it is difficult to compare the base and RLHF models on an even footing, as our methodology for sampling free-response answers likely benefits from the model’s ability to do instruction following.

Appendix C Contamination on professional and academic exams

We measure cross-contamination between our evaluation dataset and the pre-training data using substring match. Both evaluation and training data are processed by removing all spaces and symbols, keeping only characters (including numbers). For each evaluation example, we randomly select three substrings of 50 characters (or use the entire example if it’s less than 50 characters). A match is identified if any of the three sampled evaluation substrings is a substring of the processed training example. This yields a list of contaminated examples. We discard these and rerun to get uncontaminated scores.

Our filtering approach has some limitations. Our substring match can result in false negatives (if there is a small difference between the evaluation and training data) as well as false positives. We only use partial information from the evaluation examples, utilizing just the question, context, or equivalent data while ignoring answer, response, or equivalent data. In some cases, the multiple-choice options are also excluded. These exclusions may lead to an increase in false positives.

The RLHF post-training dataset is vastly smaller than the pretraining set and unlikely to have any particular question contaminated. However we did not check explicitly.

As can be seen in tables 9 and 10, contamination overall has very little effect on the reported results.

Appendix D Contamination on academic benchmarks

We measure cross-contamination between academic benchmarks and the pre-training data similarly to the methodology presented in AppendixC. Results are presented in Table11.

Appendix E GSM-8K in GPT-4 training

To improve GPT-4’s ability to do mathematical reasoning, we mixed in data from the training set of MATH and GSM-8K, two commonly studied benchmarks for mathematical reasoning in language models. The total number of tokens drawn from these math benchmarks was a tiny fraction of the overall GPT-4 training budget. When mixing in data from these math benchmarks, a portion of the training data was held back, so each individual training example may or may not have been seen by GPT-4 during training.

We conducted contamination checking to verify the test set for GSM-8K is not included in the training set (see Appendix D). We recommend interpreting the performance results reported for GPT-4 GSM-8K in Table2 as something in-between true few-shot transfer and full benchmark-specific tuning.

Appendix F Multilingual MMLU

We translated all questions and answers from MMLU[49] using Azure Translate. We used an external model to perform the translation, instead of relying on GPT-4 itself, in case the model had unrepresentative performance for its own translations. We selected a range of languages that cover different geographic regions and scripts, we show an example question taken from the astronomy category translated into Marathi, Latvian and Welsh in TableF. The translations are not perfect, in some cases losing subtle information which may hurt performance. Furthermore some translations preserve proper nouns in English, as per translation conventions, which may aid performance.

We incorporated the same MMLU prompt as[4], the model is instructed that it is an intelligent agent, supplied with the questions and a list of four answer options labelled ‘A-D’, followed by ‘Answer:’. We translate the model instruction, question and answers, however preserve the ‘Answer’ token along with the ‘A-D’ options in English. An example prompt is shown in Table12. The prompts are composed three-shot, with the three examples picked from the development set. We use three-shot evaluation over the regular five-shot because some languages map to much longer token sequences. Finally we classify the correct answer by picking the A-D token continuation with the highest probability from the model.