Let's examine two possibilities for what's going on now at law firms using Harvey, or Westlaw CoCounsel, or the like when their humans "verify" hallucination-infested AI output.
Possibility Number One: As soon as the very first legal hallucination is spotted by a human, the AI output is trashed and the humans sideline the AI for this particular task. Nobody spends any time looking at the corrupt output for even more hallucinations -- it's just ignored as slop.
Possibility Number Two: Even after the first legal hallucination is spotted and purged, the humans keep "verifying" and seek to find even more bad output -- as many judges and their clerks have done.
In neither scenario would the AI itself have the capacity to reach out to a human for counsel, at its own initiative, because it doesn't yet know that it needs help.
The "bit" here is useful, because it helps illustrate that fact that the AI agent has no concept of what it doesn't know.
Without that awareness, it will have neither the sense of timing nor the ability to prioritize that prompts humans to proactively reach out to lawyers for help -- before harm is done.
And lawyers won't be able to help AI agents with damage control, after the damage is done. As we see now at Open AI, that's a human burden.
A lawyer serving an AI agent needs to know when to tell the agent "You don't belong here at all" -- just as law firms are learning, the hard way, when "verifying" outrageous AI output.
To be sure, I was not even thinking of hallucinations-checking when I worked on this ! But more generally of agents acting to achieve whatever task there is.
Your two-possibility test is exactly the verification problem — and I'd add a third scenario most firms actually live in: they never check at all, because the output looks confident and the deadline is now. Firms that verify properly are the minority; firms that trust-and-sign are the silent majority. Which is why I've stopped arguing about whether to verify, and started arguing about where the verification should happen: inside the firm's own environment, on the draft before it leaves the building, not in the browser tab where the AI already did its thinking. When the working layer is controlled, every "human in the loop" actually sees the whole loop. That's the design decision that will matter more than any model's self-awareness.
The "lawyer as a right" framing is the sharpest part of this — and it points at something law firms are already living through. Privilege was invented so a client could make cheap prophylactic checks before doing the thing, not just mount a defense after. The firms I work with are hitting the same problem in miniature: their AI does the work, a human verifies the work, but nothing in that loop forces a check BEFORE the first draft gets written. That's the escalation path agents don't have either — and neither, honestly, do most associates. The human-in-the-loop pattern is overdue for a redesign regardless of whether the loop contains lawyers or machines.
Like the article. I do try and keep up with what AI is currently up to. I’m an older user raised on Asimov’s three- now four- laws of robotics. Have any of these ai models agreed to that?
There is already a mandatory escalation channel in EU law, and it points away from the agent. Article 73 of the AI Act has applied since 2 August 2026 and requires providers of high-risk systems to report a serious incident to the market surveillance authority within 15 days of awareness, two days for a widespread infringement, ten where a person dies. The clock starts when the provider or the deployer becomes aware, so the duty attaches to whoever was watching. That is the structural tension with a confidant model: mandatory reporting and legal privilege pull in opposite directions.
Let's examine two possibilities for what's going on now at law firms using Harvey, or Westlaw CoCounsel, or the like when their humans "verify" hallucination-infested AI output.
Possibility Number One: As soon as the very first legal hallucination is spotted by a human, the AI output is trashed and the humans sideline the AI for this particular task. Nobody spends any time looking at the corrupt output for even more hallucinations -- it's just ignored as slop.
Possibility Number Two: Even after the first legal hallucination is spotted and purged, the humans keep "verifying" and seek to find even more bad output -- as many judges and their clerks have done.
In neither scenario would the AI itself have the capacity to reach out to a human for counsel, at its own initiative, because it doesn't yet know that it needs help.
The "bit" here is useful, because it helps illustrate that fact that the AI agent has no concept of what it doesn't know.
Without that awareness, it will have neither the sense of timing nor the ability to prioritize that prompts humans to proactively reach out to lawyers for help -- before harm is done.
And lawyers won't be able to help AI agents with damage control, after the damage is done. As we see now at Open AI, that's a human burden.
A lawyer serving an AI agent needs to know when to tell the agent "You don't belong here at all" -- just as law firms are learning, the hard way, when "verifying" outrageous AI output.
To be sure, I was not even thinking of hallucinations-checking when I worked on this ! But more generally of agents acting to achieve whatever task there is.
Your two-possibility test is exactly the verification problem — and I'd add a third scenario most firms actually live in: they never check at all, because the output looks confident and the deadline is now. Firms that verify properly are the minority; firms that trust-and-sign are the silent majority. Which is why I've stopped arguing about whether to verify, and started arguing about where the verification should happen: inside the firm's own environment, on the draft before it leaves the building, not in the browser tab where the AI already did its thinking. When the working layer is controlled, every "human in the loop" actually sees the whole loop. That's the design decision that will matter more than any model's self-awareness.
The "lawyer as a right" framing is the sharpest part of this — and it points at something law firms are already living through. Privilege was invented so a client could make cheap prophylactic checks before doing the thing, not just mount a defense after. The firms I work with are hitting the same problem in miniature: their AI does the work, a human verifies the work, but nothing in that loop forces a check BEFORE the first draft gets written. That's the escalation path agents don't have either — and neither, honestly, do most associates. The human-in-the-loop pattern is overdue for a redesign regardless of whether the loop contains lawyers or machines.
Like the article. I do try and keep up with what AI is currently up to. I’m an older user raised on Asimov’s three- now four- laws of robotics. Have any of these ai models agreed to that?
There is already a mandatory escalation channel in EU law, and it points away from the agent. Article 73 of the AI Act has applied since 2 August 2026 and requires providers of high-risk systems to report a serious incident to the market surveillance authority within 15 days of awareness, two days for a widespread infringement, ten where a person dies. The clock starts when the provider or the deployer becomes aware, so the duty attaches to whoever was watching. That is the structural tension with a confidant model: mandatory reporting and legal privilege pull in opposite directions.