keepcalculator

06 Numbers and Powers

Generative AI Safeguards Businesses Should Require

Card No. 06·78255 · Updated · Standard formula, shown below

Before deploying generative AI, businesses should require documented controls for data privacy, security, accuracy, human oversight, testing, and accountability. The safeguards should match the system’s risks and intended use, with stronger review whenever AI can affect people, expose sensitive information, or take consequential actions.

Start With a Defined Use and Risk Review

A safeguard program begins by specifying what the system is allowed to do, who will use it, and what decisions it may influence. “Use AI to improve productivity” is too broad to govern. A workable scope might allow employees to draft internal meeting summaries from approved documents, while prohibiting the system from making hiring decisions or sending external messages without review.

Before a pilot, record the purpose, users, data involved, expected benefits, foreseeable harms, and the person accountable for the deployment. Review whether a conventional search tool, rules-based automation, or no new system would meet the need with less risk. Classify uses by consequence: an assistant that rewrites public marketing copy presents different risks from one that summarizes customer records or supports decisions about access to services.

  • Define permitted and prohibited uses, including whether outputs may be copied into customer-facing or operational systems.
  • Identify affected groups, sensitive contexts, and possible consequences of inaccurate or biased output.
  • Set approval gates for pilots, production release, and material changes to the model or its connected tools.

This review should be revisited when the purpose, data, model, user population, or degree of automation changes. A low-risk internal trial can become much higher risk if employees begin relying on its answers as authoritative.

Protect Data and Control Access

Employees should know which information they may enter, where prompts and outputs are stored, who can access them, and whether a provider uses them to improve its services. Do not assume that a tool is private because it is paid for, available through a company account, or described as enterprise-ready. Verify the applicable settings and contractual terms.

Set data rules before use. For example, a team drafting a product announcement may be allowed to use approved public material but not unreleased financial results, customer identifiers, credentials, or confidential source code. Where sensitive content is necessary, use an approved environment with appropriate access, retention, and processing controls. Masking or removing identifiers can reduce exposure, though it does not automatically make information anonymous.

  • Use role-based access and grant only the permissions needed for each task.
  • Set retention and deletion rules for prompts, uploaded files, generated outputs, and logs.
  • Review provider terms, subprocessors, data location and transfer arrangements, and breach-notification processes.
  • Keep secrets such as passwords, API keys, and private credentials out of prompts.

Connected applications need particular care. A model that can search company files, send email, or update records may expose more than a text-only assistant. Limit its access, separate approval for sensitive actions, and record what it does.

Secure the System and Its Supply Chain

Generative AI inherits familiar technology risks and adds new ways for untrusted content to influence behavior. Security review should cover the model provider, the application built around it, data stores, plug-ins, and any tools the model can call. A trustworthy model alone cannot compensate for an application with excessive permissions or weak authentication.

Threat modeling should consider prompt injection: instructions hidden in a document, web page, or other content may try to override the user’s request or persuade a connected system to reveal data or take an action. Treat model output and retrieved content as untrusted input. Keep access checks outside the model, validate tool requests in ordinary application code, and require user confirmation for consequential actions. Do not rely on a prompt telling the model to ignore malicious instructions as the sole defense.

Require secure development practices such as vulnerability testing, dependency review, encryption where appropriate, access logging, and an incident-response plan. Ask providers how they handle security testing, vulnerabilities, service disruptions, and significant product changes. Reassess the supply chain when a provider adds new models, features, or subprocessors.

Test Accuracy, Bias, and Reliability

Generative systems can produce fluent statements that are false, incomplete, or unsupported. They may also behave differently across wording, languages, user groups, and unusual cases. A successful demonstration is not evidence that a system is reliable in daily use.

Test against representative tasks and realistic failure cases before release. For a document-summary assistant, compare its summaries with the source documents, check whether it omits important qualifications, and test files with conflicting or missing information. For a support tool, test questions outside its approved scope and confirm it directs users to a person rather than inventing policy.

  • Measure errors that matter to the task, not just general user satisfaction.
  • Check performance across relevant groups and languages where the system will be used.
  • Test for fabricated citations, stale information, unsafe instructions, and exposure of restricted data.
  • Set release thresholds and a process for correcting failures; do not treat a single benchmark as proof of safety.

Where answers need to be grounded in internal or authoritative sources, show citations or links and make it possible to verify them. Retrieval can improve access to current documents, but it does not guarantee that the model interpreted them correctly. Human fact-checking remains necessary for high-impact material.

Keep People Accountable and Informed

A business should name an owner responsible for the system, with clear roles for security, privacy, legal, compliance, and operational teams as relevant. Employees need practical training on approved tools, data restrictions, verification, and escalation. A policy that merely says “use good judgment” does not explain what to do when an answer looks plausible but cannot be checked.

Decide when a qualified person must review outputs before they are acted on or shared. Human review should be meaningful: reviewers need the source material, enough time, relevant expertise, and authority to reject or correct the result. Simply asking someone to click approve can create automation bias—the tendency to accept a machine’s recommendation without adequate scrutiny.

Tell users when AI materially contributes to an interaction or output where that fact could affect their understanding or choices. Keep records appropriate to the risk: the model and version, approved purpose, test results, relevant settings, reviewer decisions, and incidents. Monitoring should look for drift, new failure patterns, unexpected usage, and complaints. Establish a way to pause or roll back the deployment if safeguards fail.

Governance, Contracts, and Ongoing Review

Written governance turns safeguards into repeatable decisions. Maintain an inventory of AI systems, their owners, purposes, data categories, providers, and risk reviews. Require procurement review before teams enter sensitive data into an unfamiliar service, and make clear who approves exceptions. Organizations operating across jurisdictions should have qualified advisers assess which laws and sector-specific rules apply; requirements vary by location and use.

Contracts and provider assessments should address data use, confidentiality, retention, deletion, security controls, incident notice, audit information, service changes, and responsibilities for handling complaints or correction requests. A provider’s marketing language is not a substitute for reviewing the actual terms and technical configuration.

Generative AI can also carry costs beyond subscription or usage fees: integration, testing, staff training, monitoring, human review, and incident response all require resources. Set a budget and success measures that include quality and risk, not only speed or volume. Recheck the case for deployment periodically; if the system cannot meet its accuracy, security, or oversight requirements, restrict it or stop using it.

Frequently Asked Questions

What is the most important safeguard before using generative AI?
There is no single control that fits every use. Start by defining the permitted purpose and data, then assess the consequences of errors and apply privacy, security, testing, and human-review controls proportionate to that risk.

Can employees put confidential information into an AI tool?
Only if the organization has approved that specific tool and verified its data handling, access, retention, and contractual protections. Otherwise, employees should not enter confidential information, personal data, credentials, or unreleased material.

Does human review prevent AI mistakes?
No. Review can catch errors only when reviewers have relevant expertise, access to source material, and authority to change the result. High-stakes outputs may need independent verification rather than a quick approval.

How should a business test an AI system?
Use representative real-world tasks and deliberately test likely failure cases, including incorrect answers, missing context, biased results, restricted-data exposure, and attempts to trigger unauthorized actions. Define acceptable performance before launch and monitor it afterward.

Is a generative AI provider responsible for business compliance?
Provider controls and contracts can support a business, but they do not remove the organization’s need to assess its own use, data, users, and applicable obligations. Responsibility depends on the circumstances and relevant rules.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →