Can Your Team Put Customer or Patient Data Into Generative AI? Drawing the Line

The question is already being asked — just not to you

Somewhere in your organization, someone has already pasted a chunk of work into ChatGPT. A support agent cleaned up a reply to an annoyed customer. A nurse drafted a referral letter. A sales rep summarized a long email thread. None of them meant any harm — generative AI is genuinely useful, and that is exactly why it spreads through a workplace faster than any policy can keep up.

The uncomfortable truth is that the decision about whether your team can put customer or patient data into generative AI has, in practice, already been made — by individual employees, one prompt at a time. The only real question left is whether your organization draws a deliberate, defensible line, or lets that line get drawn by accident.

This post is about drawing it on purpose. Not with a blanket ban that everyone quietly ignores, but with a framework your people can actually reason about when they have a real document in front of them and a deadline looming.

Why "just don't" doesn't hold the line

The instinct of many privacy and security teams is to ban consumer AI tools outright. It feels safe. It is also, almost always, ineffective.

Blanket bans fail for three predictable reasons, and understanding them is the first step toward a policy that survives contact with reality.

• Shadow usage. When a tool is useful and forbidden, people don't stop using it — they stop telling you they use it. You lose the one thing a privacy program depends on: visibility.
• No nuance. "Don't put data into AI" treats a public marketing FAQ the same as a patient's mental-health history. Staff know those aren't the same, so a rule that pretends otherwise loses credibility.
• It ignores the real risk. The danger was never the technology itself. It was sending personal information to a third party, on terms you haven't reviewed, that may use your inputs to train models or retain them indefinitely. That risk already exists in dozens of SaaS tools you use — generative AI just made it visible.

The line is about the data, not the tool

Here is the mental shift that makes everything else easier: the rules should attach to the sensitivity of the information, not to which app is open. The same paragraph might be perfectly fine in one tool and a reportable breach in another, depending entirely on the contract behind it.

A workable framework sorts the information your team handles into a few tiers and pairs each tier with a clear instruction. Adapt the labels to your environment, but the shape tends to look like this.

• Green — public or non-sensitive. Marketing copy, published content, general questions, anonymous brainstorming, code with no secrets. Fine in approved tools. This is where AI delivers most of its everyday value, so make this lane wide and obvious.
• Amber — internal or business-confidential. Internal strategy, unreleased plans, financials, non-personal vendor details. Allowed only in an enterprise tool whose contract prohibits training on your data and sets clear data-handling commitments. Never in a free consumer account.
• Red — personal information about identifiable people. Customer names tied to behaviour, employee records, anything that could embarrass or harm a person if it leaked. Allowed only in a vetted, contracted environment after a privacy assessment — and often not at all.
• Black — special-category and regulated data. Health information, financial account data, government identifiers, children's data, and information held under specific statutes. The default answer is no. Exceptions require a formal privacy assessment and, frequently, a tool that never leaves your controlled environment.

Healthcare and patient data: a much sharper line

If your organization touches health information, the framework above tightens considerably. Under Ontario's Personal Health Information Protection Act (PHIPA) and equivalent provincial health-privacy statutes, health information custodians carry specific obligations about how personal health information is collected, used, disclosed, and shared with service providers or agents. Broader privacy laws — Quebec's Law 25 for private-sector organizations, and the federal PIPEDA — add their own consent, accountability, and transfer-to-third-party requirements on top.

Pasting a patient's details into a consumer chatbot is, in most cases, a disclosure to a third party with no agreement, no purpose limitation, and no assurance about retention or secondary use. That is the kind of thing that turns into a breach notification, not a productivity win.

There are legitimate, compliant ways to use AI in clinical settings — but they look nothing like an employee quietly using a personal account. They involve contracted vendors, written data-handling commitments, and a documented assessment of the specific tool and workflow. If your team is asking about AI scribes or AI-assisted documentation, treat that as a formal project with a privacy review attached, not a tool someone can self-serve.

How to actually decide: a five-minute test

Policies live or die on whether a busy person can apply them in the moment. Give your team a short sequence they can run in their head before they hit enter.

• Whose information is this? If it identifies a real person — a customer, patient, or employee — slow down. If it is purely your own ideas or public facts, you are likely in the green lane.
• Could this person be harmed or embarrassed if it leaked? Health, finances, performance issues, and anything sensitive push you toward red or black, no matter how routine it feels.
• Which tool am I about to use? An approved enterprise instance with a real contract is a different world from a free personal account. The same text can be fine in one and prohibited in the other.
• Do I actually need the real data? Most of the value comes from structure, tone, and reasoning — not the specific names and numbers. Redacting or genericizing the input often removes the risk entirely while keeping the benefit.
• If I'm unsure, who do I ask? There must be a fast, blame-free way to check. A question to your privacy lead should take minutes, not require a meeting.

What the organization owes its people

A framework is only fair if the organization holds up its end. Telling staff what they can't do without giving them a safe way to do their jobs is how you manufacture shadow usage.

The practical groundwork looks like this: stand up at least one approved, contracted AI tool so there is a sanctioned lane; write the policy in plain language with concrete examples rather than abstractions; and run short, real-scenario training so people recognize their own daily situations in the rules. Pair that with a privacy assessment process quick enough to actually use, so "let me check" doesn't become "forget it, I'll just paste it in."

This is also where a clear owner matters. Whether that is an internal privacy officer or an outsourced virtual privacy officer, someone needs to own the tool inventory, the assessments, and the answer when an employee asks, "Can I use this for this?"

Draw the line before someone else does

Generative AI is not going away, and the organizations that handle it well won't be the ones that banned it hardest. They will be the ones that gave their people a clear, sensible line and a safe lane to work in.

Start with the data, not the tool. Make the green lane genuinely usable, lock down the black lane without apology, and put a fast decision process in the middle. Do that, and the question "can I put this into AI?" stops being a source of anxiety and becomes a thirty-second judgment your team can make with confidence.

If you're not sure where to start, the two most common first moves are writing a real AI use policy before more shadow usage accumulates and — for healthcare teams — assessing AI documentation tools properly before anyone goes live.

Related reading

• Do you need an AI policy before employees use chatgpt
• Can you use AI scribes in healthcare while protecting PHI