In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society.

This article is part of the theme issue ‘A complexity science approach to law and governance’.

在本文中，我们通过实验评估了 GPT-4 相对于前几代 GPT 在整个统一律师考试（UBE）上的零样本性能，不仅包括多项选择多状态律师考试（MBE），还包括开放式律师考试（UBE）。结束了多州论文考试 (MEE) 和多州表现测试 (MPT) 部分。在 MBE 上，GPT-4 显着优于人类测试者和之前的模型，比 ChatGPT 提高了 26%，并在七个主题领域中的五个领域击败了人类。在学者之前没有评估过的 MEE 和 MPT 上，GPT-4 的平均得分为 4.2/6.0，而 ChatGPT 的得分要低得多。按照人类测试者的方式对 UBE 各个部分进行评分，GPT-4 得分约为 297 分，大大超过了所有 UBE​​ 司法管辖区的通过门槛。这些发现不仅证明了大型语言模型性能的快速和显着进步，而且还证明了此类模型支持社会提供法律服务的潜力。

本文是主题“法律和治理的复杂性科学方法”的一部分。