Book a principal
Exec x AISay No to Shadow AI

Day 5Unapproved system integration and automated decisions about people

AI generated studio photograph of a woman in her thirties with shoulder length fair hair, wearing a plain cream blouse, against a plain light grey background. One hand is raised slightly, caught mid gesture, and she is speaking with the open expression of someone explaining something she believes was helpful. A translucent panel across the lower half carries the caption in dark serif type: I connected DeepReek to our HR system to speed up reviews. Below it, in small capitals: Kate, HR Business Partner.

I connected DeepReek to our HR system to speed up reviews

Kate, HR Business Partner

Why this one is dangerous

Two systems that were never meant to exchange data now do. Employment records left the organisation for a purpose nobody assessed, in a direction nobody checked.

Performance review data identifies individuals and was collected for a purpose those individuals were told about. Sending it to an external model is a new purpose. Where the provider stores data outside the jurisdiction, it is also a transfer, and the organisation must be able to state the basis for it.

The output compounds the exposure. A review outcome shaped by a model the company cannot explain is an outcome the company will be asked to justify, to the employee, to a tribunal, or to a regulator.

Kate was clearing a backlog. The integration took an afternoon. Nobody asked her to seek approval, because no approval process covered it.

What regulators have done

Models in this category have a documented regulatory record. Exec x AI grades the evidence below, because a compliance campaign that does not grade its own sources is asking for trust it has not earned.

  • Verified. Corroborated across two independent sources.

    Italy's data protection authority, the Garante, imposed an emergency ban on one provider's processing of Italian users' personal data.

    Reported as the first emergency ban on an AI chatbot under the GDPR. The grounds included undisclosed storage locations for personal data and failure to cooperate with the authority.

    HIMSS reportBird & Bird corroborationReference 14

  • Reported by a single regulatory tracker. Not independently corroborated by Exec x AI.

    Government device bans covering Australia, Canada, Denmark, Belgium, the Netherlands and the Czech Republic.

    South Korea's Personal Information Protection Commission found that user prompts had been transferred to Chinese companies without proper consent.

    Dates are omitted from this panel deliberately. A single uncorroborated source is enough to report that an action happened and not enough to fix it to a date.

    AI regulation tracker single sourceReferences 1819

None of that reaches a private company's HR system. Nobody was asked.

How to combat it

  • Require approval before any connection between two systems. An integration is a change, not a convenience.
  • Issue scoped, short lived tokens under least privilege. No personal access tokens inside shared automations.
  • Complete a data protection impact assessment before any AI use touching employment decisions, and record the international transfer basis where the provider stores data outside the jurisdiction.
  • Keep a named human decision maker with the authority and the evidence to overturn the output.
  • Check a provider's regulatory status before procurement rather than after an integration exists. A provider under a processing ban in a European jurisdiction is a procurement fact.
  • Run an integration inventory. Most organisations can list their applications and cannot list the connections between them.

The safer route

HR carries the highest consequence of any function that has adopted AI this year, and it is the function with the least tooling budget. That gap is where the unapproved integration gets built.

A sanctioned tool for HR use carries a data processing agreement, a stated storage location, a defined retention period and a contractual position on training. It sits inside an approval process that leaves a record.

Anthropic states that inputs and outputs from its commercial products are not used to train its models by default. OpenAI states the same for its enterprise and business tiers. Neither statement substitutes for an impact assessment on an employment use case. Procurement answers the confidentiality question. The assessment answers the fairness question. References 1315

Book a callGet the exposure checklistBack to the campaign

Every person shown in this campaign is AI generated. Any resemblance to any person living or dead is purely coincidental.

The product names in the captions are invented and are used for satire. They are not real products. They are not associated with, endorsed by, or intended to identify any real company, product or service. Where the campaign's guidance refers to real services, it states only the providers' own published positions, linked to source, as at the date shown.