Running an LLM on your own hardware means the math is local, not that your data is safe. Five concrete places it still leaks or gets altered (unauthenticated localhost ports, pickle-based weights, swap files, telemetry, and shared GPU memory) and how to close each one.
A deep read: the full picture, with the receipts.
Running a large language model on your own hardware feels like a fortress of solitude. The weights sit on your disk, the tokens come out of your own GPU, and nothing leaves the building. That feeling is mostly true, and the "mostly" is where you get hurt.
"Local" describes where the math runs. It does not describe your threat model. The moment you pull a model off the internet, open a port so an app can talk to it, and write a conversation to disk, you have re-created most of the exposure you were trying to escape, just with fewer people watching. This piece walks the actual path your data takes when you run Ollama, LM Studio, llama.cpp, or Jan, and points at the specific places it can leak or be tampered with. Five of them, plus the controls that actually close each gap.
What's Actually Phoning Home: A Network Audit of Local LLM Runtimes#
The first myth to kill is that "local" means "offline." It doesn't, not by default.
None of that is sinister, but it is network traffic, and "it checked for an update" is a very different security statement from "nothing left the machine." Each of these is individually defensible, and collectively it means a "local" retrieval stack can be talking to three or four external endpoints before you have typed a word.
The takeaway is not "these tools are spyware." It is that offline is a configuration you have to choose and then verify, with a firewall rule or a packet capture, not a property you get for free by running things on your own hardware.
The Open Door on Localhost: DNS Rebinding and the Loopback Myth#
Local runtimes expose an HTTP API so your apps can talk to them. Ollama listens on 127.0.0.1:11434; LM Studio and llama.cpp's server listen on ports of their own. Binding to loopback feels safe (127.0.0.0/8 is, by definition, reachable only from the same machine), so most of these servers ship with no authentication at all. Anything that can reach the port gets full access to the model, whatever context is loaded, and, in Ollama's case, the ability to pull or delete models.
"Anything that can reach the port" is a bigger set than it looks. Two ways in.
The correct mental model: a local model server is an unauthenticated service. Treat the port the way you would treat an open database port, because that is what it is.
Dirty Weights: Supply-Chain Integrity and the Pickle Problem#
You download a model (a Llama, Qwen, DeepSeek, Gemma, or Mistral checkpoint) from Hugging Face, the Ollama registry, or a random GitHub release. What did you actually just run?
The sharp risk here is not poisoned training data, since you are running inference and not training. It is the file format.
So "it's just weights" is not the same as "it's safe to load."
Underneath all of this is a plain integrity question: almost none of these downloads are cryptographically signed to a publisher you trust. You are relying on the distribution platform not being compromised and not serving you a swapped file. A hash published next to the model only helps if the attacker did not also control the page that lists the hash. Verify the checksum the maintainer publishes, prefer signed releases where they exist, and be honest that for most local models today the trust root is "the Hub didn't get owned."
Where Your Prompts Actually Live: Disk, Swap, and Log Hygiene#
Everything you type has to land somewhere. When you chat with a local model through a front-end (Open WebUI, LM Studio, Jan, GPT4All, AnythingLLM), that conversation is almost always persisted so you can scroll back to it. The question is where, and in what state.
"It's on my machine" and "it's protected" are not the same claim.
Shared Silicon: GPU Memory, Multi-Tenancy, and Side Channels#
"Local" quietly assumes the machine is yours alone. Plenty of the time it isn't: a shared workstation, a family PC, a rented cloud GPU, a container on a host you share with strangers. The moment the hardware is shared, you inherit its isolation weaknesses.
The GPU is the sharp edge, and on a shared box the risks stack.
The lesson generalizes: GPU memory isolation is younger and thinner than the CPU and OS isolation you take for granted, and a container boundary is not automatically a GPU boundary. If your threat model includes the other tenants of the machine, "local" has bought you very little on its own.
The Hardening Playbook: Controls That Actually Close the Gaps#
None of this means don't run models locally. It means treat the setup like the small production system it is. Concretely:
Running a model on your own hardware is a real privacy win: it takes the single biggest actor, the cloud provider, out of the loop. But "local" is a statement about where computation happens, not a security guarantee. The data still travels a path: off the internet onto your disk, through an unauthenticated port, into GPU memory, and back out to a logfile.
Secure the path, not the feeling.
Assume the port is exposed, assume the weights are untrusted until verified, assume the disk remembers, and configure accordingly.
Discussion