AI & Law Stuff
#20 Prompt injections, AI bans in law school, and "good enough"
Adversarial Pleadings
How does one win a case ?
You are litigating a matter, which stands to be decided by a judge or panel. On first principles, you would expect to simply do your best at writing a persuasive (at least, to your standards) brief, put together a strong (at least, to your standards) oral argument, identify key evidence or legal points that you (knowing your case) consider salient.
But beyond the legal merits, you also take care of non-legal aspects: filing on time, being courteous to the other side and the clerk, styling your prose, or, even, respecting page limits.
And then, if you happen to be a regular in front of a particular bench, it probably helps that you know what the adjudicator will like, appreciate, and adapt to their style and preferences. These might be idiosyncratic but, at the end of the day, you mean to please the person(s) in front of you, expecting that this may tilt the balance in your favour in a close case. “The good lawyer knows the law, the great lawyer knows the judge”, etc.1
The broader point is that you will, and are expected to, act strategically to prevail, within the confines of what’s permitted by the rules, good faith, and other standards of propriety.
These confines change in the age of AI and AI adjudicators, and some attorneys, at last, are trying to test them. Brazilian law firm Cescon Barrieu reports (via):
The use of artificial intelligence in the legal context has entered a new chapter following a decision issued by the 3rd Labor Court of Parauapebas, in the State of Pará, in a labor claim in which the insertion of a hidden instruction in a court filing was identified.
When processing the document through Galileu, an artificial intelligence tool developed by the Regional Labor Court of the 4th Region and rolled out nationally by the Superior Council of Labor Justice, the system detected excerpts that were not visible to the human reader, inserted in white font on a white background. The hidden content was directed at the artificial intelligence tool itself and sought to induce a possible superficial response to the filing, without challenging the documents submitted.
This is a typical “prompt injection”, that engineers keep repeating - rightly - is a key vulnerability of AI systems. As with everything, these can be done well or, as happened in this case, rather crudely:
In the same vein, the LegalQuants Red Team recently provided a further, more sophisticated proof of concept, leveraging a custom-made font that appears normal to the human eye but cannot be processed by LLMs.
And this puts the deployment of legal AI in stark perspective. So far, it has been seen mostly as a question of enhanced productivity for the people using it, or of access to justice in certain cases.
Yet it is also, at the same time, an additional vulnerability, further attack area for lawyers to win on non-legal grounds. And trying to sway AI-assisted adjudicative systems will always be frowned upon, and rightly so : it is tampering with the tribunal.
But what about deploying the same techniques against an opponent’s AI at the pleadings stage ? That’s a different matter: after all, they chose their tooling, and nothing in the procedural rules required them to pass your brief through a language model. If they do, they take on the risks that come with it. We discussed, last week, strategically inducing hallucinations in an opponent’s brief, but the guerrilla tactics can take many other forms, such as confusing the adverse AI, making it over-index on weak or moot arguments, or even conceding key points.
And in that world, someone will be needed to counter-attack and make sure the case is won on the right grounds. That someone, maybe, will be a lawyer.
The goals of a legal education
It is, I suspect, a natural bent of the human mind to seek reasons to justify their dislike of things. Praising ourselves as more than apes, we often reason from a strong feeling, rather than the other way round. And so it is with AI writing: many, I suspect, daily roll their eyes at the vision (or suspicion) of AI writing, and systematically think less of those who indulge in it.2
But not everyone is ready to simply confess to such hate,3 which means that, instead, we seek reasons: “using AI does not teach taste or judgment”; “it all looks and reads the same”; “why should I take time reading it if you did not take the time writing it”, etc. The fact that dislike may come first does not mean that these reasons are wrong, or miss the point. But it does explain some knee-jerk reactions against AI uses, and colours the various prohibitions erected here and there in this respect.
Anyhow, this is key background to a recent announcement by Berkeley Law of a mostly-universal ban on AI use by its students. The policy, which can be found here, is set, you guessed it, in terms of enhancing “the cognitive skills necessary to strategically deploy the technology, to critically assess its work product, and to uphold ethical obligations to clients and to the legal system”. And to that end, provides that:
Activities violating the rule include (but are not limited to):
Asking an AI tool to brainstorm a paper topic or thesis (prohibited conceptualizing)
Asking an AI tool to propose an organizational structure for a paper (prohibited outlining)
Asking an AI tool to compose a paragraph summarizing a legal rule for use in a paper (prohibited drafting)
Asking an AI tool to identify repetitive passages in a paper that should be cut (prohibited revising)
Asking an AI tool to polish a paper by correcting grammatical mistakes (prohibited editing)
Asking AI to generate an exam outline, elements of which are then used on the exam (prohibited exam use)
Asking AI to translate a paper originally written in another language into English (prohibited translating)
This calls for a few comments.
The first is the commonplace, but likely true, point that prohibition will mostly lead to shadow adoption, and that is likely worse than open use. I am no particular fan of the talking point that students “need to learn to use AI”, but openness in this respect is better than a climate of suspicions, doubts, and vague accusations that AI bans can only lead to.
The second is to query whether banning AI will in fact improve the cognitive skills necessary to use it properly. Developing the skills necessary to judge an AI work-product requires, one assumes, encounters with such work products - lest the learning be deferred to when they are on the job already, and the costs of mistake are highest. Moreover, such rules usually proceed from a baseline of “the world without AI” that can easily be seen with optimistic glasses; are we so sure law students learned critical skills in law schools before ChatGPT was released ? I am not sure.
And the third, related but more important, is in terms of low expectations and misguided universalism. Anyone teaching students sees - but maybe refuses to acknowledge - that, as with everything, they are not equal in all things, and that many are just going through the motions. You do your best to engage them but, eventually, you can enjoy the ride only with those (not always a majority) that are ready to partake in it, and make the job actually worth it. And it is these students that are the most likely to hear the advice that it’s better, for their own growth, to refrain from using AI. Berkeley Law’s list would make excellent counsel for the students worth teaching; as a blanket rule, it will miss its mark.
This points to a deeper version of the same problem, well put in a recent piece by Owen Yingling of The New Critic:
Goethe once noted, “A teacher who can arouse a feeling for one single good action, for one single good poem, accomplishes more than he who fills our memory with rows on rows of natural objects, classified with name and form.”
[…]
The best teachers — those who can stimulate students like Goethe’s romantic educator or Socrates in Meno — are […] eccentric, sometimes malicious, and occasionally downright insane. Already, the standardization of academia and teaching over the last 50 years has decimated this type. The best and worst professors at any given college are usually the aged fossils who arrived before grad-school and a tenure-track position became a narrow gate and student evaluations became gospel; no one can deny that the unthinking application of metrics and checkboxes funnels teaching standards toward mediocrity.
Perhaps, it will turn out, the issue is not even with the students, but with the teachers, who fail to give the students the motivation to become good lawyers in the first place, and to give them “l’envie d’avoir envie”. Full AI prohibitions may make sense to such teachers, but they read as a missed opportunity.
The Final Footnote
When do you know if a piece of work is good enough ?
You have been tasked with producing an artefact: some legal advice in a transaction, an oral statement for pleadings, a piece of legal research, etc. You, being sovereign over your level of commitment and efforts, can decide to make it the most important thing in your life, or just a regular Monday afternoon. But in any event, you’ll have at some point to stop working and shoot your work product. The question is then, when can / should that happen ?
Few people ever introspect about this, for the excellent reason that it is a hard question. It’s part of these things that you know when you see it, and in any event is often circumstantial. You stop because you reached a deadline or exhausted the allocated hours, because your boss/client is yelling at you, because you are tired or fed-up, or because that particular doctrinal source has run dry - and you actively choose to ignore the dozen possible alternatives that could add something new.
All this builds up to endow lawyers with a “sense” that something is ready to be submitted or not, and of how to juggle the spectrum between “too raw” and “overcooked”.
That’s one issue lurking behind a recent piece published by the ABA on “The Warning Signs of AI Dependence in Legal Practice”. The authors highlight the “hazards of frictionless work”, making the point that:
AI tools save time and reduce the friction of the blank page. This convenience creates a mental hazard. In the traditional practice of law, the “blank page” acts as a necessary obstacle. It forces a lawyer to engage in the slow, often painful labor of organizing thoughts and selecting the precise language to serve a client’s needs.
[…]
Because the machine writes with a professional tone, the lawyer may feel a false sense of momentum. This ease creates a dangerous complacency. […] True advocacy requires the tension of creation; without it, the lawyer becomes a mere passenger to the machine’s output.
But this stops where the interesting part begins. Worrying about complacency is fair enough; the point that “writing is reasoning” is worth repeating again and again, lest some forget.
But the deeper puzzle is not that we might stop too early, or even decline to start and delegate, out of laziness, but that we do not actually know what the stopping rule is - not even when we are diligent. In this context, using AI is not only about when to stop adding text to text - something we also discussed earlier - but also acquiring that sense (the ABA piece calls it “professional intuition”) that something is good enough.
And that is what makes all of this so annoying to a scientific, engineering mind.4 We would love to know when to stop, because a reliable stopping rule would be a proxy for what good looks like - and anything you can specify that precisely, you can eventually automate. By contrast, notions of “intuition”, “taste”, and “judgment” are frustratingly vague, more akin to astrology than astronomy.
I have argued before that “taste” may flatter us more than it informs, and that much of what registers as judgment is really pattern-matching from long experience, internalised conventions, or plain risk aversion - none of which is obviously beyond a machine’s reach. Knowing when you have covered enough ground is the same problem, not when to stop writing, but when to stop looking.
And yet. Perhaps the vagueness is a feature; not knowing when to stop is precisely what keeps us committed to the work. I keep saying that a virtue of automation (and now AI) is that it sheds light on why certain practices exist, and this “why” is often counter-intuitive. In this vein, not knowing when to stop forces you to stay on your guard, and that might be a good thing we risk losing.
There is some empirical evidence that experience correlates with success; see here for a recent review of the literature.
Others are enthusiastic and eager to use AI in an ever-greater array of circumstances, finding a new life in becoming an AI enthusiast - hence the manifold cringe posts on LinkedIn about how ChatGraudemini helped them win a Supreme Court case singlehandedly.
Although shoutout to Sam Kriss for confessing that he simply hates AI writing, no hedge, no particular reason.
When to stop is, in fact, one key difference I highlight when teaching coding to law students: at some point, your code runs, and that’s over: your data analysis is complete, your app is live and running, you can breathe a sigh of relief and go to bed. But most legal tasks are not like that.