Local LLM: your data stays in house
Many companies have stopped cloud AI by policy because nobody can say where the inputs end up. A language model on your own hardware answers that question with the architecture itself. The policy stays as it is.
- Local LLM
- A local LLM is a large language model that runs on your own hardware or in a dedicated environment, so that no input is sent to an external service. Business data never leaves your network, which sits comfortably with policies that forbid passing data on. The same setup is also called an on-premise LLM or a self-hosted model.
What is banned is rarely the AI. It is the data leaving.
Read your own IT policy again and you will usually find a sentence like: business data must not leave the company. Generative AI is rarely named.
What blocks the request is an open question: where does an input go, how long does it stay there, and does anyone train on it? As long as nobody can answer that, the approval sits in a drawer.
If the model runs inside your network, the architecture answers the question on its own. That is a shorter route than renegotiating the policy.
Cloud or local is the wrong first question.
Four points decide whether a setup meets your policy. If all four hold, a cloud service sometimes passes too.
Where it is processed
Physically: which country, which server. With default settings, requests end up in a different region more often than people expect.
What stays behind
Input history, logs, caches. Deleted often only means deleted in that one place.
Whether it trains on it
That is set in the contract and in the admin console. One of the two is not enough.
Who can prove it
When the auditor asks: do you have to write to the provider, or do you show your own log and you are done?
Three ways to build a local LLM
None of them is always right. It comes down to how sensitive the data is and what quality you need.
Fully local
An open model on a GPU server inside your network, served with vLLM or Ollama. Business data never leaves the network. The choice for HR files, legal and engineering data.
Dedicated environment
A dedicated environment in an EU region, contractually without training on your data. Current commercial models with a fixed place of processing.
Mixed
Sensitive data locally, general text externally. A rule decides what goes where and logs the decision. In practice the most common setup.
What changes compared with the usual setup
The same function, a different answer at the next audit.
| Point | Usual setup | Without data leaving |
|---|---|---|
| Document storage | Consolidated in an external store | Stays on your file server, only the index is built |
| Transcribing meetings | Recordings go to an external service | Runs on your own GPU, the recording stays inside |
| Model | A provider's API | Open model on your own hardware |
| Search index | External vector database | Inside your network, deletions handled by you |
| Auditor's question | Answered after asking the provider | Show your own log, immediately |
| Model changes | When the provider decides | When you decide, with a bit more effort |
Scroll sideways for all columns →
When a local LLM is the wrong choice
We do not want to sell a setup, we want to sell a system that runs. If local does not fit, we say so in the first call.
One reason often remains anyway: the request gets approved, and at the next audit you answer yourself.
Speech stays in house, in production for months
In the meeting project the speech recognition runs on the customer's own hardware. Meetings contain HR matters, prices and plans nobody is supposed to know yet. Processing them outside the network was never an option.
Name matching runs against the internal staff list. Who took on which task is nothing anyone outside needs to know.
Details are in the meeting dashboard case study. What a multilingual knowledge system looks like when index and search stay in house is shown in the knowledge agent case study.
Local LLM: common questions
Related topics
Tell us which sentence in your policy it is stuck on.
Thirty minutes are enough to say which setup gets through. If the cloud is enough, we say that too.