AI Hallucinations, Confidentiality and Lawyers’ Duties

Lawyers using generative AI must check its output rather than treat it as fact, protect client information from unauthorized disclosure, and comply with the professional duties that apply to their work. Those duties do not disappear when a task is delegated to software: the lawyer remains responsible for competent, confidential, truthful representation and for reviewing any work product before relying on it.

What an AI hallucination is—and why it matters

A generative AI system produces text by predicting likely sequences from patterns in its training and context. It may give a fluent, confident answer that contains fabricated or inaccurate material; this is commonly called a hallucination. The term does not mean the system has intentions or beliefs. It describes an output that sounds plausible but is unsupported or wrong.

In legal work, errors can take consequential forms: a nonexistent case, a quotation that does not appear in the cited opinion, an incorrect rule statement, a miscalculated deadline, or a summary that omits a qualification. A citation may point to a real decision while misrepresenting what it held. The risk is not limited to invented citations; a correct-looking answer can still be outdated, incomplete, or based on assumptions that do not fit the client’s facts.

Courts have addressed instances in which lawyers submitted filings containing fabricated authorities, and some have imposed sanctions or other consequences under the rules and circumstances involved. These examples are not a universal rule about every AI mistake. They illustrate a practical point: a lawyer who presents a proposition to a court must have a sound basis for it, regardless of how the draft was produced.

Professional duties still apply

Ethical requirements vary by jurisdiction, and lawyers should consult the rules and authoritative guidance applicable where they practice. In the United States, the ABA Model Rules offer a useful framework, but they are models rather than binding rules unless adopted by a jurisdiction. Duties commonly implicated include competence, confidentiality, communication with clients, supervision, candor to tribunals, and reasonable fees.

Competence includes understanding enough about a tool’s capabilities and limitations to use it responsibly. A lawyer need not become a machine-learning engineer, but should know, for example, that generated answers may be wrong, that systems can handle prompts differently, and that a citation or factual assertion needs independent verification. Whether to use AI, and how much review is required, depends on the task, the stakes, the client’s needs, and the lawyer’s skill.

Delegation also does not transfer responsibility. If a lawyer asks an employee, vendor, or AI service to prepare work, appropriate supervision and review remain important. A lawyer should not charge a client for time not spent or describe AI-generated work in a misleading way. Whether disclosure to a client is needed depends on the circumstances, governing rules, engagement terms, court requirements, and whether the tool materially affects the service or decision-making.

Confidentiality and data handling

Before entering client information into an AI tool, determine what happens to the information. Depending on the provider and configuration, prompts or uploaded documents may be retained, reviewed, used to improve services, or processed by other vendors. Terms differ, and product settings can change. Do not assume that a tool is confidential merely because it is paid, widely used, or described as suitable for business.

Confidentiality analysis is broader than asking whether the information is privileged. A lawyer’s duty may cover information relating to a representation even when it is not privileged, while privilege and work-product protection have their own legal tests. Sharing information with a third party can raise questions about protection, but the outcome depends on the facts and governing law. A public AI service should not be treated as a private legal workspace without reviewing its terms and safeguards.

• Review the provider’s current terms, privacy policy, retention practices, training use, access controls, and deletion options.
• Use only the minimum information needed; remove names and other identifying details where that can be done without undermining the task.
• Check whether the client’s instructions, engagement agreement, protective order, or applicable law restricts disclosure or storage.
• Use approved organizational tools and follow firm policies for access, device security, and document handling.

De-identification is not a guarantee: a combination of dates, facts, locations, or unusual circumstances can identify a person. If the risk cannot be adequately managed, do not submit the information. For sensitive matters, seek appropriate internal guidance before using an unfamiliar service.

Verification before using or filing output

Review should be proportionate to the risk. A low-stakes brainstorming outline calls for different checks from a court filing, a client-specific legal conclusion, or a deadline calculation. The more consequential the output, the more important it is to consult primary sources and confirm that the answer fits the actual record and jurisdiction.

1. Confirm every legal authority in an authoritative source. Open the decision or statute, check that it exists, and read the relevant passages rather than relying on a generated summary.
2. Verify quotations, citations, procedural history, and whether the authority remains good law using reliable research tools and applicable court rules.
3. Compare factual summaries with the source documents. Look for missing qualifiers, invented details, and confusion between allegations, evidence, and findings.
4. Recalculate dates and numerical results independently, including the assumptions and counting rules involved.
5. Make the final judgment yourself. Confirm that the advice is appropriate for this client, jurisdiction, and procedural posture.

Keep an appropriate record of the tool’s role and the checks performed when firm policy, client needs, or the matter’s risk makes documentation useful. Do not treat an AI detector as proof that text is accurate or inaccurate; such tools cannot substitute for checking sources and facts.

Policies, supervision and practical boundaries

Firms can reduce avoidable risk by establishing clear rules rather than leaving each user to guess. A practical policy identifies approved tools, prohibited data, review requirements, who may authorize new services, and how to handle errors or suspected disclosure. Training should use realistic examples, including fabricated citations and summaries that quietly omit important facts.

Supervisors should consider the user’s experience and the work’s sensitivity. Junior staff may need specific instructions to verify every authority and distinguish a first draft from a final analysis. Lawyers should also understand relevant vendor arrangements and controls; technical claims about security should be checked against the actual contract and configuration, not accepted as marketing shorthand.

AI can be useful for organizing public information, generating an initial outline, or suggesting questions for further research. It is not a substitute for legal research, professional judgment, or client communication. Nor should a lawyer assume that every use must be disclosed, or that no use needs disclosure: the answer depends on the applicable rules, court orders, agreement, and the role AI played.

Frequently Asked Questions

Can lawyers use ChatGPT or another generative AI tool?
Often they can, subject to applicable professional rules, client instructions, court requirements, and firm policy. Lawyers should evaluate confidentiality and reliability before use and independently review work product.

Is it safe to paste client information into an AI chatbot?
Not without first understanding the particular service’s terms, data handling, and approved settings. Minimize or remove identifying information where appropriate, and do not submit sensitive material if confidentiality cannot be protected.

Who is responsible if AI creates a fake legal citation?
The lawyer who uses or submits the citation remains responsible for meeting professional and procedural obligations. The consequences depend on the jurisdiction, context, and applicable rules, but verification is essential.

Do lawyers have to tell clients when they use AI?
There is no single answer for every task and jurisdiction. Consider client instructions, engagement terms, applicable rules, the significance of the tool’s role, and whether disclosure is needed for informed decisions or transparency.

Does removing a client’s name make a prompt confidential?
No. Other details may identify the client or matter, and confidentiality also depends on how the service handles submitted data. De-identification reduces some risks but does not eliminate them.

Can AI-generated legal research be filed without checking it?
No. Verify authorities and quotations in reliable primary sources, check the facts against the record, and apply professional judgment before relying on or filing the material.

Related

• Franchise Business Plan & Financial Forecast Guide
• FC 26 Set Piece FC Evolution: Best Players & Upgrades
• Encrypt Decrypt Word 2016 Document: Full Guide