Conducting an AI PIA in Healthcare: A Practical Walkthrough

When the assessment is the hard part, not the form

Most healthcare teams we work with don't struggle with the idea of a privacy impact assessment. They struggle with what happens when artificial intelligence enters the picture. A PIA template built for a new patient portal or an EMR upgrade assumes you can describe, with reasonable confidence, what data goes in, what comes out, and who touches it along the way. AI tools bend all three of those assumptions.

An AI scribe listens to a clinical encounter and generates a note. A triage model scores risk. A vendor's large language model summarizes a chart. In each case the data flows are murkier, the decision logic is harder to inspect, and the line between 'tool' and 'decision-maker' gets blurry. That is exactly why an AI PIA is worth doing properly rather than treating it as a checkbox exercise.

This walkthrough lays out how we actually run an AI PIA in a Canadian healthcare setting, step by step, in plain language. It's written for the privacy lead, clinical informatics manager, or founder who has been handed an AI project and a deadline, and needs to know what 'doing this right' looks like.

Step 1: Confirm you actually need a PIA, and scope it before you write anything

The first mistake is filling in a template before the project is understood. The second is assuming every AI tool triggers a full assessment. Some lightweight, low-risk uses can be handled with a screening or a threshold question; others clearly warrant a full PIA. Decide which path you're on first.

In healthcare, the bar for a full assessment is low because personal health information (PHI) is at stake and provincial health-privacy laws — Ontario's PHIPA, Alberta's HIA, and comparable regimes elsewhere — impose strong accountability duties on anyone handling PHI. (Where no dedicated health-privacy statute applies, public bodies fall under general public-sector laws such as British Columbia's FOIPPA, which itself requires a PIA before a program launches.) If an AI tool will process, store, or even transiently 'see' PHI, treat a full PIA as the default and justify any decision to do less.

• Map the AI use case in one or two sentences: what problem it solves, what data it consumes, what it produces.
• Identify whether PHI is involved, and whether the output feeds a clinical or administrative decision.
• Decide the assessment type: a full PIA, a PIA paired with a threat and risk assessment (TRA), or a screening that may escalate.
• Set the timeline early — a meaningful AI PIA is rarely a one-afternoon task, and rushing it is how gaps get missed.

Step 2: Map the data flows — including the ones the vendor doesn't volunteer

This is where AI assessments diverge most from traditional ones. With a conventional system you can usually draw a clean diagram of inputs, storage, and outputs. With AI you have to chase down questions vendors don't always answer on the first ask, and the answers materially change your risk picture.

Pin down where the model runs, where data lands, and what happens to it after the immediate task is done. 'The data is processed in the cloud' is not an answer; 'processed in a Canadian region, encrypted in transit and at rest, not retained after the response, and never used to train the vendor's models' is the kind of specificity you're looking for.

• Where is the model hosted, and in which jurisdiction does processing occur? Cross-border processing changes your obligations.
• Is PHI used to train or fine-tune the vendor's models? For most healthcare deployments this should be contractually off the table.
• What is retained, for how long, and who can access it — including the vendor's own staff and any sub-processors?
• Are inputs and outputs logged, and are those logs themselves a new repository of PHI you now have to protect?
• How are accuracy errors and 'hallucinations' surfaced, corrected, and recorded?

Step 3: Assess the risks unique to AI, not just the usual ones

A good AI PIA covers the familiar territory — collection, use, disclosure, consent, retention, safeguards — but it has to go further. AI introduces failure modes that traditional systems don't, and a PIA that ignores them isn't really an AI PIA.

Three categories deserve particular attention. First, accuracy and reliability: an AI scribe that quietly drops a medication or invents a symptom is a patient-safety issue as much as a privacy one. Second, transparency and explainability: can a clinician understand why the tool produced a given output, and can a patient be meaningfully informed that AI was involved? Third, automated decision-making: if the tool influences care decisions, both clinical accountability and emerging regulatory expectations come into play — Quebec's Law 25, for instance, requires organizations to inform individuals when a decision is based exclusively on automated processing.

Document each risk, its likelihood and impact, and the control that brings it to an acceptable level. Where you can't mitigate a risk, say so plainly and escalate it — an honest PIA that names a residual risk is far more useful than one that papers over it.

Step 4: Work through a worked example — an AI scribe

Make this concrete. Suppose a clinic wants to adopt an AI scribe that records consultations and drafts notes. Walking that single use case through the PIA shows how the pieces fit together.

On data flows, you confirm audio and transcripts are processed in a Canadian region, not retained after the note is generated, and not used for model training. On consent, you design how patients are told AI is being used and how they can decline. On accuracy, you require a clinician to review and sign off on every note — the AI drafts, the human remains accountable. On safeguards, you check encryption, access controls, and audit logging, and you confirm the vendor will sign appropriate data-handling terms.

The conclusion most clinics reach is that an AI scribe can be used responsibly while protecting PHI — but only with the consent design, human review, and contractual controls that the PIA forces you to think through in advance, rather than discovering after launch.

Step 5: Land the controls, mitigations, and sign-off

The output of a PIA is not the document — it's the set of decisions and commitments it produces. Each identified risk should resolve into a concrete control with an owner and a date, and the residual risks should be accepted by someone with the authority to accept them.

Tie privacy controls to contractual ones. The strongest PIA findings are toothless if they don't show up in the vendor agreement: data residency, no-training clauses, retention limits, breach-notification timelines, and sub-processor disclosure all belong in writing. The PIA is where you decide what you need; the contract is where you make it stick.

• A risk register with named owners, mitigations, and target dates.
• Contractual requirements derived directly from the PIA findings.
• A consent and transparency approach patients and clinicians can actually follow.
• A human-in-the-loop checkpoint wherever AI output touches a clinical decision.
• Formal sign-off, plus a review date — AI tools change, and so should the assessment.

Step 6: Treat the PIA as living, because the model will change

AI systems are not static. Vendors update models, add features, change where data is processed, and adjust their training practices. A PIA that was accurate at launch can drift out of date within a release cycle, so build in a trigger to revisit it — at minimum annually, and whenever the vendor materially changes the tool or its data handling.

This is also where ongoing governance pays off. Organizations that already have a basic AI governance framework and a clear AI-use policy find each new assessment faster and more consistent, because the hard thinking about principles has been done. The PIA then becomes an application of settled policy rather than a fresh argument every time.

Where this leaves you

An AI PIA in healthcare is more demanding than a traditional one, but the method is the same discipline applied with sharper questions: scope it honestly, map the real data flows, name the AI-specific risks, mitigate them with controls that make it into the contract, and keep the assessment alive as the technology shifts.

Done well, it's not a brake on innovation — it's what lets a clinic adopt genuinely useful AI tools with confidence that PHI, patients, and clinicians are protected. If you're staring at an AI project and a deadline and aren't sure where the line sits, the two questions worth answering first are when an AI PIA is actually required and whether your specific use case — like an AI scribe — can be done safely at all. The related reading below is a good place to start.

Related reading

• When do you need an AI PIA
• Can you use AI scribes in healthcare while protecting PHI