In May 2023, the Irish Data Protection Commission fined Meta 1.2 billion euros for transferring personal data of Europeans to the United States without a valid legal basis. It is one of the largest fines imposed under the GDPR, and it shares a pattern with nearly all others: the failure lies in how the data moves, not where the model is computed. This is where local AI changes the game: the model runs on your GPU, the prompt does not travel, and no new processor is added to the chain. What does not change is the Regulation: executing locally simplifies compliance, it does not eliminate it.
GDPR is not switched off just because the model lives on your server
Regulation (EU) 2016/679, known as GDPR, follows the data, not the hardware. If you process customer invoices with an open-weights model installed in your office, your company is the data controller and retains the usual obligations: security (Article 32), minimization (Article 5), and records of processing activities (Article 30).
What a local approach does solve are three typical cloud weaknesses: international transfers under Chapter V, the chain of processors under Article 28, and the exposure of data to third parties you do not control. What it does not solve are the rights of the data subject: access, rectification, erasure, objection, restriction, and portability remain operational, and you must be able to locate a customer’s trace in your conversations, logs, and backups if requested.
There is a frequent trap with pseudonymization. Replacing names with identifiers does not anonymize anything as long as a table exists that links an ID to a person: for the Regulation, that is still personal data.
The next step is inexpensive and organizes everything else: draw a processing map and note every flow where identifying data enters a prompt, an indexed document, or an audio file.
Data that should never leave your server
Start with the obvious: resumes and payrolls, support tickets with names and emails, contracts with tax IDs and addresses, and Article 9 data, which includes health, biometrics, ideology, or sexual orientation. The typical scenario that local AI cuts off at the root is the employee who pastes a sheet with thousands of contacts into a public chat to organize it: the data ends up on foreign servers without a processing agreement or documented basis.
Voice transcriptions count the same: a voice recording allows for the identification of the person speaking. And if a visual system identifies faces, the images process Article 9 data; uploading them to a cloud transcription or image generation service adds new hands to the chain.
Then there is the category that no one plans for: API keys, tokens, and internal credentials pasted into a prompt by mistake. In the cloud, they go to the provider’s server; locally, they stay in your log, which should be purged with the same discipline.
A practical rule divides your flows into three levels: local-only for identifying and Article 9 data, local-preferred for operational tasks, and cloud-with-contract for anything that identifies no one. Write down this classification: it is the first piece of your compliance document.
Where personal data hides in a local AI stack
Conversations do not live only in the chat window. Interfaces like Open WebUI store history in a database on the server, and the inference process writes logs where the full prompt may appear.
If you set up RAG, embeddings duplicate the problem. A vector stored in pgvector, Qdrant, or Chroma reproduces content from the original document, including names, so deleting the source PDF does not delete anything until you purge the collection and re-index.
Fine-tuned weights are the uncomfortable case. If you train an adapter with customer emails or real tickets, that information remains embedded in the parameters, and extracting it later is an open research problem called machine unlearning. To understand when a model trained with personal data can be considered anonymous, the EDPB published an opinion in late 2024 that is worth reading before adjusting weights with your own data.
Backups, snapshots, and telemetry also fall off the radar. Many desktop tools query their server to check for updates or send usage metrics; review which connections each component makes and disable those that provide no value.
Your inventory must list every point where something persists (history, logs, vector index, weights, backups) and define a retention period with a purge date for each. Without that inventory, the right to erasure becomes an unfulfillable promise.
When the cloud is still reasonable
Local AI is not dogma. For flows without identifying data, a provider with a processing region in the EU, a signed processor agreement, and a documented commitment not to train on your data can fit without drama.
What does not fit is doing it without paperwork: you are missing Article 28, Chapter V if there is a transfer, and, where applicable, the transfer impact assessment. The framework has changed three times in less than a decade: Schrems I struck down Safe Harbor in 2015 (C-362/14), Schrems II invalidated the Privacy Shield in July 2020 (C-311/18), and the Commission approved the new Data Privacy Framework in July 2023.
The scale of exposure is marked by the figures: serious GDPR infringements can reach up to 20 million euros or 4% of total worldwide annual turnover, whichever is higher (Article 83.5). With that range ahead, it is better to decide based on data, not intuitions.
The legal basis also does not change by running on your hardware. If you process a customer’s data to provide them with a service, Article 6.1.b (performance of a contract) usually covers it; for internal uses regarding employees, consider documented legitimate interest with a balancing test.
With that framework on the table, the decision is made per flow and per document, not by the provider’s brand: define now which flows can leave with a contract and which must never leave.
A local stack that withstands an audit
For language, Ollama or llama.cpp expose an API over 127.0.0.1; Ollama listens by default on port 11434, and that port must not be open to the internet. Open-weights models like Llama, Gemma, Qwen, or Mistral cover most internal tasks, and some, like Mistral 7B, are distributed under the Apache 2.0 license. If you prefer buttons over a terminal, LM Studio does the same with a graphical interface.
Downloading weights is not transferring personal data: your machine talks to the registry (only your IP and the model name travel). If your policy requires it, download each model once and share it via an internal mirror.
To query documents, a local RAG with pgvector, Qdrant, or Chroma and an orchestrator like LangChain works without the cloud; PrivateGPT emerged specifically for that use case. For voice, whisper.cpp and faster-whisper transcribe on your machine, so customer voice recordings do not pass through any provider.
For images, ComfyUI with SDXL or Flux generates on your GPU without uploading anything; if you want to compare numbers before deciding where each flow runs, the blog has a local image generation comparison of SD, Flux, and SDXL vs. cloud services. For video, open-weights models like Wan, LTX-Video, or Mochi follow the same logic, and we test them in detail in our local generative video guide.
On that foundation, three inexpensive controls support the entire building: disk encryption with LUKS or BitLocker, separate accounts with minimum permissions, and a firewall rule that keeps the API on the local interface.
Pilot a limited case, such as support with RAG over tickets or HR classifying resumes, on a GPU you already have. Before generalizing, decide where each piece of data lives (database, vector index, or history) and who can delete it.
The minimum that must be documented
The principle of proactive accountability (Article 5.2) does not require infinite paperwork: it requires proportional evidence. With fewer than 250 employees and occasional low-risk processing, Article 30.5 may exempt you even from the record of activities; if you process Article 9 data or if AI appears in habitual processes, document it anyway.
A data protection impact assessment (Article 35) is required when processing is high-risk, for example, with biometrics or systematic screening of people. And the 72-hour breach notification (Article 33) applies equally: ransomware on the inference machine is a notifiable breach like any other.
The location where the computation runs also does not change the obligations of the European AI Act, in force since August 2024: the use is what matters, and a local AI that screens candidates is still a high-risk system. Be careful with storage: without access control or change logs, spreadsheets are a poor layer for personal data, and a database with delimited permissions fulfills that role better both locally and in the cloud.
Your minimum checklist: legal basis per flow, retention with automatic purge, access control, destination of logs, and a real test of an erasure exercise.
Where to start this week
First, inventory the points where personal data enters today: prompts, vector indices, or audio, including the cloud services you already use. Second, mark sensitive flows as local-only and move them to a stack like the one in the previous section. Third, for what remains in the cloud, verify the region, the processing agreement, and the training policy, and leave a written record.
Local AI does not abolish the GDPR: it moves the responsibility to your infrastructure, with its attack surface, its logs, and its backups. In exchange, it turns privacy by design (Article 25) into verifiable elements: the firewall rule, the purge policy, and the processing inventory. That is the shift in perspective to start with — and the first audit to prepare for.