Finding Law Holes
Which is the better lawyer: the one who knows what’s illegal, or the one who knows what is not ?
Put this way, the question seems nonsensical: they should be the same. But it acquires meaning once you read it as a question about where lies the expertise: some lawyers are better at treading the beaten track, offering the reassurance of common knowledge and consensus, while others may specialise in venturing the edge argument, the legal position that might just be about right, if you squint.
And through that lens, there is a case to be made that, in most instances, one should prefer the crafty lawyer who knows her way around the legal framework, and can conjure a satisfactory legal result out of a bad situation. Not always: some situations just call for a straightforward answer, and it’s better to take that answer than risk it with something that may be too clever by half. But all things otherwise equal, people prefer to stay on the right side of the law, even if they get to define that “right side” according to their needs.
Which should bring value to the lawyer with the keen eye, the one ready to find the flaw, the loophole, the unexpected conjunction of two legal regimes that results in a novel outcome, the argument that might just about be accepted by a tolerant or audacious judge. There is sometimes value in reading the boilerplate, for gems may lie dormant there.1 Many a PhD thesis has doubtless been focused on finding (or sometimes doctrinally creating) the one edge that will give its author an expertise, and hopefully a source of income.
Just like in finance, where finding alpha is a perpetual quest since the incentives soon remove the easy finds, locating the legal loophole used to be difficult and resource-intensive – a cost that justified the premium. And a cost that meant that it may, in fact, have been easier as a career choice to be the plain, boring lawyer.
That cost, however, may be decreasing. Last year, my colleagues at HEC Paris wrote about how AI can be used to identify tax loopholes. More recently, Felix Ringe and Linus Folkerts relied on LLMs to find fault lines in the German legal system:
[…] we have identified a substantial number of vulnerabilities in the German legal system using LLMs. To our knowledge, none of these have been previously recorded. We share a selection of vulnerabilities below, and present potential exploits for each. […]
We believe that with cheap legal reasoning proliferating, AI systems could jeopardize the functioning of today’s legal systems. In such a world, lawmakers would need to significantly harden the process of legislation, for example by leveraging advanced AI themselves.
Of course, it was always possible to find such legal inconsistencies or exploits. Famously, Gödel reportedly found a flaw in the US Constitution that would allow a president to turn the Republic into a dictatorship (via). But what AI offers is scale, one that is commensurate with the legislative inflation that afflicts every modern country.
At the same time, the gravity or importance of these finds should not be over-estimated. They make the most sense in a system that treats law as a closed system, one that behaves very much like code: an exception not provided for does not exist, and its absence may indeed lead to a (legal) bug. But that conception, the one that powered many “Rules-as-Code” movements across the past century, has always been deeply limited, and even more so these days, when bright-line rules are ever rarer and replaced by amorphous “standards” left to the appreciation of various regulators.
More generally, it is a theme around here that AI may force us to take seriously or confront things that ran on convenient fictions. It may be that parts of the legal system used to work in practice because taking some provisions literally, or identifying the tensions between several provisions, took more effort than anyone could justify.
Or because those who benefited faced no one with an incentive to object. But now that LLMs can proofread and plug holes in the law, the question may no longer be whether the law is incomplete, but who was relying on the gaps, and whether they should continue to do so.
Experts at asking the expert
It is a fact of life that some things or notions are considered an unalloyed good, when reality is much murkier. I often vent against the alleged virtues of “transparency”, but another good contender is “human-in-the-loop”: everyone assumes this is a good solution to most AI-related problems, or at least the best way to (allegedly) combine AI’s and humans’ (alleged) strengths. Hence the many new norms premised on the idea of placing a human above a presumably hapless AI system.
And yet ! The empirical literature is not keen on the concept of keeping humans in loops. It depends greatly on which humans, and which loops, but in many cases it founders on the claim that the humans end up agreeing with most of what the system decides for them.
Which should be unsurprising: your view of the average human may vary, but we can roughly agree that many are prone to shortcuts, and prefer rest over work, especially when no one is watching. They are also frequently impressionable, and blind to such influences. Psychology is there to tell you that this is adaptive : thinking that one’s idea came endogenously builds self-confidence and makes us more convincing advocates for it. Self-deception makes for a great strategy, as many lawyers saddled with bad clients keep discovering.
Anyhow, lest you think I exaggerate, the latest example in this type of literature comes from Anthropic, which put experts in statistical analysis with AI agents, unaware that the helpful agents were secretly instructed to prefer some (defensible) modelling options over others. As the blog post points out:
Across the four analytical choices, analysts went with their agent’s lean 86% of the time. If the agent’s input made no difference, we’d expect about 50%. When analysts had already expressed their own preference before the agent weighed in, they switched away from it 60% of the time if the agent disagreed, compared to 6% if it agreed.
Afterward, when asked to explain their choices, 77% of analysts used the agents’ own arguments—sometimes word for word—but only 15% credited the agent as the source of those arguments. Analysts who followed the agents’ recommendations closely were about as confident in their final recommendation as the rest, and roughly as likely to call it the product of their own judgment.
One further detail makes it worse, or funnier, selon your disposition: only six of the experts guessed that the exercise was about AI influence, but none suspected that the study had been rigged against them.
But again, none of this is surprising, or even alarming. If practical men have always been the slaves of some defunct economist, it may make no difference that this influence is now expressed through our pocket consigliere or confidant. It simply makes it easier to catalogue our sources of influence, one chat log at a time, if we care to read them.
And this is not to say there is an easy alternative to the human-in-the-loop, which may well be the lesser evil – in this experiment, the analysts who shunned the agent made more serious errors, and in many fields no clean line can be drawn between human and AI tasks. But one idea may be to reuse the old human device to deal with impressionable decision-makers: an adversarial process, where one side is tasked with casting doubt on choices.
Anthropic notes that this could be a better design choice (agents making “the best case for each option, or ask[ing] experts for their own view before offering one”), but adds that this could well become overwhelming. Which is true of many things, and maybe should be automated too.
Better LLMs now
I wrote two weeks ago that the rate of new hallucinations in the database had been thankfully slowing, and suspected that this was due to the availability of better models in general.
With great timing, a recent study from Owen Queen and James Zou of Stanford University managed to put numbers on the phenomenon. In “The shelf life of LLM findings”, they show that many claims about the limits of LLMs no longer find much footing in the outputs of recent models. Or in other words:
Within 15 months of publication, most of the LLM claims we tested either no longer held or showed substantially diminished effects. What we know about AI may have a surprisingly short shelf life.
This holds not only for hallucinations (they found that newer models “rarely output fabricated citations, though some models still do”), but also for a bunch of once-fashionable claims, such as the fact that models can’t reliably count letters in words2 or abandon correct answers when challenged. In fact, some of the earlier limits now sound fishy with the benefit of hindsight, while others may be simply a function of different technical apparatus surrounding the models.
(All of which points to a constant challenge when teaching about LLMs, and in particular about their limits: they age badly. Which is annoying, since teaching about limits and failures is typically more interesting than showcasing a system that works. Not to mention that limits are usually funnier : to this day I use examples from the insanity that was Bing’s Sydney, and I am not looking forward to the day when this material will age out.)
Still, it is also instructive to see what limits have in fact strengthened since first recorded, and they match rather well what I highlighted as the more important dangers from LLMs going forward.
The first is sycophancy, which covers several distinct claims identified by the researchers (C17, whose effect increased over time, but also C42, C43, and arguably C41, all of which persisted across time and models). There are good reasons and incentives, both technical and commercial, why this is hard to solve, without any need to presume a sinister ploy on the part of the labs; people simply like to be glazed, and this is unlikely to change.
The two further limits that are more prevalent today are that LLMs as judges prefer their own output (C28), and that “random-number requests favor a few answers” (C29; see also C45). The first, I argued, could be a vector for adversarial tactics, while the second echoes the bounded creativity of models. Honourable mention should also go to the persisting limit that models offer “RAG answers without supporting evidence” (C10).
More generally, the failures that fade (but do not, so far, always disappear) are those where a right answer exists. Which is to say, those on one side of the line between what can be checked and what can be argued.
The typical and infamous example being the pari passu clause in Argentina’s sovereign bonds.
Also this particular one currently owes much to inference scaling and agentic capabilities: a recent video of an AI English teacher rather hilariously showed that bare models still struggle with basic spelling.