Warning: Advice Against Deploying AI Representative in Legal Proceedings
In an unprecedented move, Jerome Dewald, an AI entrepreneur, represented himself in a court case against insurance firm MassMutual Metro New York using an AI avatar. The event, which took place on March 26, 2025, has sparked controversy and raised questions about the legitimacy of using AI in legal proceedings.
Dewald, who heads an AI startup called Pro Se Pro, admitted that he should have provided a heads-up that he was going to use AI to present his case. The court was not informed that the video submitted by Dewald was not of him but an AI-generated video of a handsomely generic man, named "Jim."
Associate Justice Sallie Manzanet-Daniels expressed disapproval and felt misled by Dewald's actions. She questioned Dewald's inability to articulate his case due to health issues or deception. Dewald, who was diagnosed with throat cancer 25 years ago and has ongoing health issues that make continuous speaking challenging, used a stock AI replica from Tavus for his court appearance. However, it was clarified that Dewald's AI representative, "Jim," was not created with his own platform, Pro Se Pro.
The use of AI in legal proceedings is a contentious issue. AI-generated materials often have errors or fabrications, such as false legal citations or quotes, known as "AI hallucinations," which can undermine the integrity of legal filings if not carefully vetted. There have been cases where filings including AI-generated falsehoods caused delays, embarrassment for lawyers, and judicial reprimands, demonstrating courts’ intolerance for misleading or inaccurate submissions.
Submissions containing AI-generated content must be independently verified for accuracy. Failure to disclose the use of AI or submitting AI-generated false information may lead to sanctions, fines, or criminal consequences, as seen in specific jurisdictions like under Pennsylvania’s law, where knowingly submitting false information carries penalties.
Transparency about AI use and careful human oversight is critical when employing AI tools in legal contexts. While AI can assist legal professionals (e.g., automating paperwork or research), the legal responsibility and representation must remain with qualified humans.
Pro Se Pro, Dewald's AI-focused startup, has been at a standstill for about a year due to lack of funding. The company aims to help people represent themselves in legal matters with AI tools. However, the recent court incident has cast doubts on the legitimacy and ethical implications of using AI avatars in court.
The incident serves as a reminder that while AI technology continues to evolve, its use in legal proceedings must be approached with caution and transparency to maintain the integrity of the judicial system.
Technology has raised questions about its place in legal proceedings, particularly after the use of an AI avatar in court by Jerome Dewald, an AI entrepreneur. The future implementation of artificial intelligence in courts may face challenges due to concerns over accuracy, lighting inadvertent errors or fabrications in AI-generated materials, such as legal citations or quotes.
Gizmodo reported on the controversy surrounding Dewald's AI court appearance, highlighting the importance of transparency and human oversight when employing such technologies in legal contexts. Dewald's start-up, Pro Se Pro, is focused on using technology to help individuals navigate legal matters themselves, but the recent court incident has sparked debate about the ethical implications of using AI avatars in legal representation.
Given the potential for AI-generated falsehoods to sow confusion and question the legitimacy of legal proceedings, tech companies must ensure that their AI tools can be trusted to uphold the standards of truth and justice in the courtroom, rather than posing a threat to them.
