Anthropic’s Claude, OpenAI’s ChatGPT and Alphabet’s Gemini may be great for helping with research or creating spreadsheets, but one thing they decidedly are not is your attorney. If anything, turning to artificial intelligence for legal advice can wind up backfiring, as it did for a white collar criminal defendant in New York who tried to shield from federal prosecutors his communications with Claude–the ones where he asked it for a legal defense strategy.
It seems obvious that you shouldn’t rely on artificial intelligence for legal advice, particularly if something important, like your freedom or a lot of money, is at stake. What’s new is that even discussing legal matters with your favorite chatbot could be risky.
In February, U.S. District Court Judge Jed S. Rakoff of the Southern District of New York ruled that AI’s large language models are not attorneys, and any communications with them can be used against criminal defendants just like any other non-privileged documents. The judge noted in his decision that his ruling “appears to answer a question of first impression nationwide”—meaning no other courts in the U.S. have ruled on this issue. His conclusion on privilege would presumably apply to civil litigation too.
The privilege question arose in a criminal securities fraud case against Texas businessman Bradley Heppner, the former chairman and CEO of publicly-traded GWG Holdings, who was charged with defrauding investors out of more than $150 million. Heppner, 60, was convicted by a jury this month and is awaiting sentencing. He faces a maximum of 20 years in jail.
According to court records, after Heppner learned last year that he was being targeted by federal investigators, he began asking Claude to outline a “defense strategy” he might be able to use in court against criminal charges.
When FBI agents later raided his home and seized a number of electronic devices, they came across 31 saved copies of documents and conversations that Heppner had had with Claude about his criminal case. After Heppner’s lawyers asserted that those documents were privileged, since he’d had those conversations in preparation for talking to his lawyers, the government segregated them. It then asked Rakoff to rule on whether they were in fact protected from the government’s inspection under either the attorney-client privilege or the work product doctrine.
On Feb. 17, Rakoff ruled they were not. In his 12-page order, Rakoff backed up his finding with a number of legal arguments.
First and foremost, Rakoff pointed out, “Because Claude is not an attorney … that alone disposes of Heppner’s claims of privilege.”
Not only that, Rakoff wrote, but he noted that any criminal defendant confidentiality – such as with attorneys – requires a “trusting human relationship.”
“No such relationship exists, or could exist, between an AI user and a platform such as Claude,” Rakoff wrote.
The judge went on to point out that Claude’s parent company, Anthropic, specifically notes in the Claude terms of use – which Heppner agreed to – that it retains the right to all “data,” including user conversations with Claude, which can then be turned over to authorities “in connection with claims, disputes or litigation.”
Rakoff went on to observe that the only possible way Heppner’s communications with Claude might have been kept confidential is if his attorney specifically directed him to ask Claude for legal advice, and in that case it might be argued that Claude was acting as a lawyer’s agent. But in this instance, Heppner did so on his own initiative, and his attorney only learned about his AI conversations after the fact.
Moreover, when prosecutors asked Claude during the case if it could provide legal advice to users like Heppner, it responded, “I’m not a lawyer and can’t provide formal legal advice.” It then recommended contacting a licensed attorney.
The ruling provides a “stark warning” to business owners who may have already made use of AI large language models for various legal strategies, New York attorney David Feldman opined on his firm’s website. “The decision draws a clear distinction between human legal counsel and digital assistants,” he wrote, noting that the primary takeaway is that Heppner’s use of Claude could now be weaponized against him by federal prosecutors. “Lawyers should consider warning their clients that the use of open AI (systems) may not be considered confidential communication,” Feldman wrote.
Feldman said in a follow up email to Forbes that he believes this is a growing problem among C-suite executives who think AI will or can replace expensive attorneys. “I tend to spend too much of my time cleaning up messes created by this type of action,” he wrote. “With the advent of AI, clients are even more emboldened to try to handle things without our advice.”
Alan Pentz, a small business consultant who’s been focused on AI development in recent years, said the news was unsurprising, given that legally, chats with AI large language models are “all discoverable” in court. In other words, executives should take care not to share sensitive information with large language models such as Claude, ChatGPT, Gemini or others.
“AI is just like Google searches, documents, and email,” Pentz wrote in an email to Forbes. “Proceed accordingly.”
The ruling is also emblematic of a new reality for businesses of all sizes, said Samir Black, the founder of Georgia-based tech firms Data Pros Consulting and Forseti Verify.
“It’s the new angle of attack… this is very much in line with the new age of investigation,” Black said of memorialized AI conversations such as Heppner’s. “It’s no longer about the files on your machine. It’s the conversations with AI and chatbots.”
Want to be more successful? Subscribe to the weekly Forbes Careers newsletter to get insider tips and insights.
More on Forbes:
