How do you assess the privacy and security risk of an AI vendor?

What should an AI vendor risk assessment actually cover?

An AI vendor assessment is a standard third-party risk review extended with questions unique to machine learning: what data the model sees, whether your data trains it, how outputs are generated, and who is accountable when the model is wrong. Treat any AI tool that touches personal, health, or confidential information as a high-risk processor and scope the review accordingly.

Work through these dimensions in order so you can stop early if a deal-breaker surfaces:

• Data scope and sensitivity: what categories of data the tool ingests (personal information, PHI, financial, IP), and whether that data is necessary for the use case or scope creep you can trim.
• Data use and training: whether your prompts, inputs, and outputs are used to train, fine-tune, or improve the vendor's models, and whether you can opt out by default.
• Hosting and data residency: where data is processed and stored, which cloud regions and which sub-processors and model providers are involved, and whether data leaves Canada.
• Security posture: independent evidence such as a SOC 2 Type II report, ISO 27001 certification, recent penetration tests, encryption in transit and at rest, and access controls.
• AI-specific risks: model accuracy and bias, hallucination and the consequences of a wrong output, human oversight, explainability, and retention of prompts and generated content.
• Contracts and accountability: a data processing agreement, breach notification commitments, audit rights, deletion on termination, and clear liability for AI-driven harm.

How do you tell if an AI vendor trains on your data?

Read the terms, then confirm in writing, because the default behaviour differs sharply between consumer and enterprise products. Many free or consumer AI tools reserve the right to use your inputs to improve their models, while paid enterprise and API tiers from the same provider often exclude customer data from training by default. The tier you actually use determines the answer, not the brand on the box.

Ask the vendor directly: are my prompts, uploaded files, and the model's outputs used to train or fine-tune any model? Is training opt-out the default or something I must request? How long are prompts and outputs retained, and are they reviewed by humans for quality or abuse monitoring? Get the answers in the contract or DPA rather than a sales email, since marketing claims and binding terms frequently diverge.

This matters most when staff might paste personal information, patient details, source code, or unreleased strategy into a tool. If those inputs feed a shared model, you have effectively disclosed that information to a third party, and potentially exposed it to other customers through model outputs.

What security evidence should you ask an AI vendor for?

Ask for independent, current evidence rather than self-attestation, and read it rather than just collecting it. A logo on a trust page is not proof; the underlying report and its scope are what matter. Request the following and check that each genuinely covers the product you are buying:

• A SOC 2 Type II report (read the scope, the period covered, and any exceptions the auditor noted) or an ISO 27001 certificate with its Statement of Applicability.
• Recent third-party penetration test summaries and how findings were remediated.
• A current sub-processor list, including which foundation-model providers and cloud regions are used.
• Encryption details for data in transit and at rest, plus tenant isolation in multi-tenant environments.
• Access controls, including SSO, role-based access, and audit logging you can review.
• Breach history and incident response commitments, including notification timelines.

When does an AI vendor review need a formal AI Privacy Impact Assessment?

Run a formal AI Privacy Impact Assessment (AI-PIA) whenever the tool processes personal or health information, makes or materially informs decisions about individuals, or introduces new automated profiling. A questionnaire tells you what a vendor claims; a structured assessment tells you whether using that vendor with your data is actually defensible.

An AI-PIA extends the traditional PIA method. According to the federal Office of the Privacy Commissioner, a PIA is a risk management process that helps institutions ensure they meet legislative requirements and identify the impacts their programs and activities will have on individuals' privacy, analysed against core privacy principles such as limiting collection, retention, accuracy, safeguards, and limiting use and disclosure. An AI-PIA layers on the questions a model raises: what the system was trained on, how decisions are explained, where bias or inaccuracy could harm someone, and what human review sits between the model and the affected person.

Most private-sector businesses are not legally bound by the government PIA mandates that bind federal and provincial public bodies, but the same methodology is best practice and is increasingly demanded by enterprise, healthcare, and government buyers, and by regimes such as PHIPA and Quebec's Law 25. Doing the assessment before you deploy is far cheaper than unwinding a tool already wired into your workflow.

What contract terms protect you when using an AI vendor?

The contract is where verbal assurances become enforceable, so push the AI-specific protections into the data processing agreement and master terms before signing. The strongest technical posture is undermined by a contract that lets the vendor change the rules later or disclaims all responsibility for the model's output.

• A data processing agreement that names the purposes data may be used for and explicitly prohibits training on your data unless you opt in.
• Sub-processor transparency, with notice and the right to object before new sub-processors or model providers are added.
• Data residency and cross-border commitments aligned to your obligations under Canadian privacy law.
• Breach notification within a defined, short timeframe, with cooperation on investigation and regulator or individual notice.
• Deletion and return of data on termination, including any copies held for model improvement.
• Audit or assurance rights, and clear allocation of liability for harm caused by inaccurate or biased outputs.