[{"source":"Scoop.it - Documentation juridique (Stéphan Cottin)","url":"https://verfassungsblog.de/the-currently-uneven-intelligence-of-ai-in-comparative-constitutional-law/","titre":"The (Currently) Uneven Intelligence of AI in Comparative Constitutional Law","id":3218421,"pubDate":"Thu, 15 Oct 2026 08:26:22 +0200","description":"This article belongs to the debate » Thinking Machines, Thinking Lawyers14 October 2026The (Currently) Uneven Intelligence of AI in Comparative Constitutional Law From Years of Research to a Single Prompt? Artificial intelligence (AI) has the potential to accelerate and improve the quality of research. This potential has inspired numerous projects exploring how AI could help scholarship on constitutional law. For example, there is research using AI to interpret constitutions,1) apply constitutional principles,2) and predict court rulings.3) Even the question of whether AI could improve the process of constitution-making has been studied.4) AI’s potential is particularly significant for comparative constitutional research. Systematically comparing legal systems requires locating sources from different jurisdictions, which are often difficult to access and only available in local languages. The creation of comparative legal datasets has thus involved considerable personnel, time, and financial resources. AI can overcome many of these obstacles. AI models can search for and translate foreign-language sources, filter them, and convert the filtered information into a dataset. In doing so, AI offers the prospect of datasets — which previously required years of research and the efforts of numerous experts and research assistants to compile — being just a prompt away. However, whether AI models can generate comparative law datasets that are qualitatively comparable to human-coded datasets has not yet been addressed (although Guha et al provide an example for a broader legal benchmarking exercise).5) We therefore conducted our own benchmarking study.6) We found evidence suggesting that recent AI models may be able to code simple factual data, but they are likely unreliable substitutes for expert human coders when conceptual or interpretive judgment is required. The Benchmarking Project: Seven Coding Tasks, Four Datasets, Two LLMs In our project, we tested the ability of two large language models (LLMs) to code different types of information relevant to comparative constitutional law. We specifically assessed whether LLMs can reproduce information from four existing comparative constitutional law datasets, all of which were compiled through human coding and which served as a ‘ground truth’ for our exercise. From these datasets, we asked the LLMs to code seven variables, which can be thought of as seven tasks relevant to the creation of comparative constitutional law datasets. Each of these tasks is either a test of fact-finding or reasoning abilities. First, from the Executive Term Limit Evasion Dataset,7) we drew information on whether presidents in 106 presidential systems have evaded constitutionally mandated term limits between 2000 and 2019. The dataset includes one variable on whether presidents attempt to overstay past the end of their term. The second variable captures what strategies executives use for doing so, such as amending the constitution, replacing it with a new one, or asking the court to interpret term limits out of the constitution. While determining whether a president overstayed is a fact-finding task, assessing the strategy used to do so requires reasoning. Second, from the Comparative Constitution Making Dataset,8) we drew information on the process by which constitutions were adopted in democracies between 1900 and 2015. This dataset also includes fact-finding variables such as (1) whether citizens were involved in the formulation, discussion, or submission of reform proposals, and (2) whether citizens participated in popular referendums at any stage in the process. The dataset’s most important variable (3) captures whether constitutions reflect ‘plural elite agreement,’ that is, whether all relevant elite factions in society were involved and represented during the process. The recreation of this variable requires reasoning. Third, from the Characteristics of National Constitutions Dataset9), we drew on its coding of 43 first-generation constitutional"}]