Lawyers today are facing a problem they did not anticipate when generative AI tools first arrived: Their own clients are using these tools to second-guess them, and in doing so, may be handing over confidential case information to systems that cannot be trusted to keep it.
That is the warning from Kay Firth-Butterfield, CEO of Good Tech Advisory and joint head of AI at Doughty Street Chambers in London, one of the world’s most prominent human rights chambers. She is also the former inaugural head of AI and machine learning at the World Economic Forum, a TIME100 Impact Award recipient, and author of “Coexisting with AI: Work, Love and Play in a Changing World.”
“We are already seeing it,” Firth-Butterfield said in an interview with The AI Innovator. “There is going to be a big issue about attorney-client privilege.”
The concern is specific. Clients, unaware of the implications, are feeding not only their own case documents but also opposing counsel’s papers into large language models to get a second opinion on their lawyers’ advice.
“What does that do to confidentiality?” asked Firth-Butterfield.
One person who sought AI as second counsel explained that the chatbot was her boyfriend — and that it therefore felt natural to ask it what it thought. “The machine is very capable of being persuasive, of being manipulative, of nudging us in different ways,” Firth-Butterfield said.
She noted that surveys also suggest 20% of American men now describe being in an intimate relationship with an AI. Noting that education is needed, Firth-Butterfield predicts that “as more people become enmeshed between human and machine, we’re going to see more of those problems.”
In that context, a client treating a chatbot as a confidential advisor is not an edge case — it is a pattern that law firms need to address before it becomes a liability.
What attorneys should do now
Firth-Butterfield’s prescription for attorneys is to treat the first client engagement as an opportunity to set expectations, in the same way a physician sets boundaries around internet research. She recalled that after a breast cancer diagnosis three years ago, her oncologist told her that using the internet was acceptable — but provided a list of trusted resources. Attorneys, she said, should do the same.
“If you put everything into this machine, you might be in contempt of court, you lose all of your own privacy, and we might lose client-attorney privilege,” she said. “Those sorts of things are important at the very beginning, and it’s really on us as lawyers to do it.”
She also called for law firms to establish internal governance boards — standing committees that include the managing partner, the chief technology officer, the risk and compliance team, the hiring team and human resources — with collective responsibility for where AI is used, how employees are trained to use it and how results are checked.
“You should have an internal board across your entire law firm,” she said. “All of those people need to understand where AI is being used and how to train every employee in how to use it and how to use it wisely.” For solo practitioners and small firms without the resources for a formal board, she said the imperative is to seek out trusted information sources.
Additionally, Firth-Butterfield noted, “we have to be employing humans who are capable of checking that the AI is doing the right thing.”
She emphasized that lawyers need to learn about AI so they can recognize when it is hallucinating and avoid bringing those hallucinations into court, because failing to do so fundamentally undermines the entire legal system. Even legal AI products designed specifically for lawyers still carry high hallucination rates, which means checking the AI’s output is not optional — it is a professional obligation. “I advise a lot of companies, and all my companies have good training going on, because you can’t have wise use of artificial intelligence unless your employees are trained,” she said.
“Just because you’re not using an LLM doesn’t protect you from having to check your work,” Firth-Butterfield said. On vendor selection, she said, “I think that one of the problems is that people don’t know the questions to ask of vendors. And so, if they don’t know the right questions, then they should admit that they don’t know the right questions and get a professional to ask those questions.”
The deepfake evidence problem
Firth-Butterfield flagged another front opening in litigation: deepfaked evidence. Such evidence includes deepfaked medical reports and images of accidents.
“Think about being an insurance company and having all those claims made and some of them will be deepfakes and some of them won’t,” she said. Attorneys, she noted, are not equipped to distinguish AI-generated fakes from authentic documents.
For attorneys looking for guidance on AI and the law, she pointed to the work of Amal Clooney — also a member of Doughty Street Chambers — who co-founded the Oxford Institute of Technology and Justice (OITJ) in October 2025, a collaboration between Oxford University’s Blavatnik School of Government and the Clooney Foundation for Justice. The OITJ focuses on expanding access to justice through AI, accountability for unlawful cyber operations and protecting human rights in AI-assisted court proceedings. Its work includes tracking the use of AI in courtrooms across fifteen jurisdictions.
A court draws a line
Firth-Butterfield cited as the most consequential development a June 2025 ruling from England’s King’s Bench Division that she said every lawyer practicing in the United Kingdom is now bound to follow.
The case — R (Ayinde) v London Borough of Haringey; Al-Haroun v Qatar National Bank, [2025] EWHC 1383 (Admin) — was decided by Dame Victoria Sharp, president of the King’s Bench Division. It arose from instances of lawyers using generative AI to produce legal arguments and witness statements without verifying the output, resulting in false citations and fabricated case references being submitted to the court.
“They have truly fundamentally understood the threat to the human right to remedy if the common law is undermined by hallucinations,” Firth-Butterfield said of the court’s judgment.
The judgment makes clear that the duty to verify AI-generated research rests on the attorney — not the tool, and that those in leadership positions such as heads of chambers and managing partners bear personal responsibility for ensuring compliance. The court’s sanctions range from public admonition and costs orders to contempt proceedings, criminal referral for perverting the course of justice, and, in the most egregious cases, life imprisonment.
“They put the onus for hallucinations back on the lawyer,” Firth-Butterfield said.
She connected that judicial position to a tension built into how AI has been marketed to the legal profession.
“We have been sold AI as something that allows us to get rid of humans and will make us quicker. But if you’ve got to check everything that your AI produces, then it doesn’t get rid of humans and it doesn’t make us quicker,” she said.
Governance gaps and global models
On regulation, Firth-Butterfield pointed to the European Union AI Act as the clearest framework currently in force. Within the United States, she said the Colorado AI Act — which takes effect later in 2026 and is modeled on the EU framework — represents a promising approach because it starts from the question of which human values are at risk, rather than enumerating specific technologies to regulate.
She also cited the Singapore Model AI Framework, which began in her offices during her tenure at the World Economic Forum, as an example of a governance approach that balances policy and technology without being prescriptive. In contrast, she noted that the United States has frameworks — including the NIST AI Risk Management Framework — but no federal mandate requiring their use.
“I think when we have this conversation we should do what the Europeans do, which is say, ‘Which values do we hold so fundamentally that make us this nation or this area or this region?’ and then say, ‘How will AI interfere? What are the risks that we have from AI in regard to that?’”
Looking ahead, Firth-Butterfield said 2026 will accelerate a pipeline problem with which law firm leaders have not fully reckoned. If firms rely on AI to handle the work that junior lawyers would otherwise do, they will not develop the next generation of partners — and that structural gap will become apparent within a decade.
“Will law firms understand that if they don’t take young lawyers in, they have no partners for the future?” she asked.
Be First to Comment