Setting the terms
Is there truth in law ? Well, one sort of truth certainly: some legal debates get definitively resolved one way or the other, be it in a court, in parliament, or even (though rarely) through doctrinal conversations.
But that truth is often rightly seen as soft: dependent on circumstances, contingent on specific patterns, and liable to be reopened by anyone with the right incentives and amount of free time. And in fact, some legal questions live in an in-between state, settled for now, but with everyone knowing that at some later point – after a change of personnel, say – this may easily change this state of affairs, such that confirmations of the settlement just occasion a shrug and a “see you next time”. Insert your favourite thing to relitigate to death here, but you see the point.
As anyone arguing for a living (or for a hobby) will know, debates can be framed, and that framing has an impact on what kind of arguments, participants, and especially what kind of solutions can be aired. Framing can change, but it tends to be sticky. So, whoever sets the terms of the debate already acquires an edge – especially if one’s contribution is the only accessible or retrievable contribution to the debate.
You see where I am going with this. In the paper on Adversarial Inputs, I described the potential exploit consisting in populating a data void with your views, expecting the other side’s AI to retrieve these views and be influenced by, well, you. That could certainly be useful !
And I am mentioning this because of a real-life example I encountered recently: France is, as is known, currently having some budgetary issues. This is a financial crisis, of course, but it also includes a legal issue: if, as seems likely, the government eventually adopts a budget by ordinance – something unprecedented – then nobody is entirely sure who (between (a) the Council of State and (b) the Constitutional Council – yes, these are two different bodies) is competent to review it.
There is a legal debate about this. But it played out in large part within doctrinal journals, most of which are hopelessly paywalled. And, because the question is so new and suddenly urgent, the contributions are limited in number.
With one notable exception: eminent constitutional jurist Denys de Béchillon wrote a much-noticed (and very accessible) blog post on the question here, opining rather straightforwardly that the Council of State is competent.
Now, what would an AI model answer when put that question ? You guessed it, the Council of State, relying primarily on that blog post.
This was not always the answer, however: some models, especially when prompted directly in French, would note that the debate remains open, but they would always rely substantially on de Béchillon’s blog post. Note, too, that in most of my attempts, the models would also cite a rival blog post that concludes in favour of the competence of the Constitutional Council, but rarely use it beyond providing background material. Which is to say: models can cite exactly the same source, and yet come to different conclusions, putting a question mark on what “well-sourced” even means.1
Two key points to convey about adversarial inputs are that, first, much of this has antecedents (there is an interesting 2019 report on the relevance of filling data voids), and second, that the boundary between innocent conduct and manipulation of AI is hard to discern – if it can even be identified. Or in other words, what one may see as a (helpful) contribution to the debate may also be described as a (manipulative) form of jurisquatting, the attempt to pre-empt a subject by being there first, and fill the data void with AI retrievers and readers in mind.
(Some potential factors to weigh: (A) interest, so as to distinguish between scholars opining in the abstract and counsel in pending matters – disclosures are meant for humans, and may not work as well for AI readers. (B) timing, since a post written into a void, before the debate has really started, does more work than one joining a crowded field. And (C) status, on the basis that with great authority comes great responsibility – a state administration publishing a FAQ or explanatory notes may stack the deck for models trained to prefer authoritative sources.)
Yet this at least suggests one likely development, which echoes last week’s post on writing for AI: expect legal contributions to be increasingly publicly available, so as to better inform AI models that rely on open sources, maybe even blogs, to come to a conclusion. Or see the epistemic environment fragment depending on who or what has access to which paywall. Maybe both.
Laisser-Passer A83
At some level, an administration can be seen as a machine that produces and handles documents. Type “bureaucrat” on Google Images and what you see2 is, sure, the eponymous bureau, but more often than not one buried under binders and sheets of paper. This illustrates, as I have argued, that the “lump of law” is never fixed: another set of guidelines can always be updated or enacted, another form improved or streamlined.
It should now be obvious that AI is turbocharging all this, and bringing its own deficiencies into the process. EU outlet Rapporteur reports:
The EU likes to think it is at the cutting edge of regulating AI.
But its own institutions in Brussels are struggling to cope with the torrent of activity the technology has unleashed.
[…]
Translators in the Parliament, who adapt amendments and bills into 24 official languages and use machine translation but not AI themselves, have stumbled across ‘hallucinations’ – references to non-existent laws in legal texts drafted by MEPs.
At times, the legal service has also been tied up for days investigating claims made by MEPs using AI.
Good fun, and fully expected, as is the fact that parliamentary questions are increasingly asked, but also (allegedly) answered, by AI, in “a largely human-free conversation”. And one that fits the preference of these institutions for ever more text.
But less recognised is that the reverse is true too: documents produce administration. When you start being flooded with text, or when that text requires precise structure, the basic human mind cannot be the measure of it all, and so we start to build processes to help with the flow.
Which is why this bit from Ethan Mollick’s recent work with Fable 5.2, in which he set the model to create distinct “worlds”, is not even surprising.
In the first two days, the agents built 328 unique worlds. […] A few days later, I noticed that instead of hundreds of new worlds a day, it was making about a dozen. I asked a few agents to explore what had happened, and they found something surprising.
The AI had built a bureaucracy.
The agents decided that my request to “polish” existing worlds meant each new version had to beat the old one in a blind vote by a group of judge agents on twelve test screenshots. […] Soon, launching a new world required an approved idea, three AI judges and a full code review against the rule book. All told, that took about a day. Reading the rule book alone used up a large part of an agent’s working context.
Mollick sent an agent and found bureaux. More than a century ago, French philosopher Alain had this to say about the French state’s disastrous decision to enter war in 1914: “There responsibility crumbles away. We look for a man, we find bureaux. We look for a decision, and find a circulaire.”
I am not sure what is worse. When Alain wrote this, you at least eventually found a clerk, who may well be amiable, not a log file or a vote among judge agents. On the other hand, AI administrations may be the most legible ever built: you may be able to identify what happened in minutes. But legibility is not responsibility, and the distinction may matter more than we think.
AI Laws are coming for you
A new bill was recently introduced in the New York Senate, S10741, and aims at “[p]rohibit[ing] judges from using fabricated legal authority in adjudicative writings”. Shortly put, the bill would make it misconduct in office for judges to “knowingly or recklessly include[], rel[y] upon, or adopt[] material fabricated legal authority in adjudicative writing”. The avowed goal is to ensure “Judicial AI Accountability and Integrity”.
I think it is a very badly thought-out piece of legislation, and that it may even be counterproductive.
To my mind, the main risk is not judges “knowingly” relying on hallucinated authorities (why would they ever do that ?), but judges relying on counsel’s citations on the assumption that they are good law. The key question is thus how “recklessly” is applied in practice. Set the threshold too high, and the law adds little (New York already has AI-specific judicial guidance: Judicial Ethics Opinion 26-133); too low, and it risks imposing a considerable burden on judges and their clerks, one they may lack the resources to meet. A burden that potentially includes, in my view, some degree of interference in how judges work.
Of course, judges are expected to check the authorities they rely upon. But the adversarial process also serves that purpose: parties debate what the law is, and judges have always relied on that debate. The proposed standard, I am concerned, could well replace this dynamic with a climate of generalised suspicion that would do no one any good.
More precisely, judges (and in turn, parties) may simply respond by citing fewer authorities or relying on a narrower set of trusted sources. Or judges might also explain less and be more defensive in their reasoning, expecting that disappointed litigants will recast disagreement about precedent as disciplinary complaints. I am not sure that would improve the quality of legal argument or judicial decisions. (I said parties would cite less: some may even cite more, hoping to overburden the court and expecting a mistake they can exploit on appeal.)
Nor do I think the safeguards adequately address these concerns. Saying that AI use alone is not misconduct does not explain how much verification is enough. Protecting good-faith interpretations of unsettled law does not necessarily protect an honest mistake about an authority. A clearer assurance that reasonable verification or an honest mistake will not amount to misconduct would have been more useful.
Anyhow, last time we discussed a bill here – SB 574 – I likewise opined that it was duplicative and unnecessary. It passed, however, and will soon be “good” law. But if legislators only passed perfect laws, there would be no need for us lawyers.
Open question whereas the style and length differences between the posts explain their varying reception by AI.
Besides a very confusingly named manga, which I declined to investigate further.



