Private LLM for Business: When Self-Hosted AI Makes Sense (and When It Doesn’t)

Many companies want the productivity of ChatGPT or Claude but hesitate at the same point: where does our data go? Contracts, patient notes, financial records and source code are exactly the documents where AI is most useful — and exactly the ones legal and compliance teams don’t want leaving the building. A private LLM for business is the answer many teams reach for. But “private” can mean several different setups, with very different costs.

This guide explains the options, when self-hosting is worth it, what it really costs, and how to decide without overbuilding.

What “private LLM” actually means

There are four common setups, from least to most control:

1. Business plans of public AI tools. Enterprise tiers of ChatGPT, Claude or Gemini typically commit not to train on your data and offer admin controls. Your data still leaves your infrastructure.
2. Model APIs with contractual protections. Calling a model through an API (directly or via a cloud provider such as AWS, Azure or Google Cloud) with a data processing agreement, regional hosting where available and zero or limited retention options.
3. Open-weight models in your own cloud. You run a model such as gpt-oss, Llama, Mistral or Qwen on GPU servers in your own cloud account. Prompts and documents never reach a model vendor.
4. Fully on-premise. The same open-weight models running on hardware you own, sometimes fully air-gapped.

Options 1 and 2 are “private” in a contractual sense. Options 3 and 4 are private in a technical sense: nobody outside your organisation processes the data.

Why self-hosting became realistic

Two years ago, open models were noticeably behind the best hosted ones. That gap has narrowed enough to matter for business tasks. A clear signal: in August 2025 OpenAI released gpt-oss, two open-weight models under the Apache 2.0 licence. According to OpenAI, gpt-oss-120b runs within 80 GB of GPU memory and gpt-oss-20b needs only 16 GB — meaning a capable model fits on a single data-centre GPU, and a smaller one on a workstation.

Mature serving tools such as vLLM, Ollama and llama.cpp have also made running these models an engineering task rather than a research project.

When a private LLM for business makes sense

• Regulated data. Healthcare, legal, finance, HR and public sector data where policy or regulation restricts third-party processing.
• Contractual obligations. Client contracts that forbid sending their data to external processors.
• Trade secrets and source code. R&D documents, pricing models, unreleased products.
• High, predictable volume. Classifying or extracting from hundreds of thousands of documents a month, where per-token API fees add up and a fixed GPU cost becomes cheaper.
• Offline or restricted networks. Factories, ships, field operations or secure facilities.

When it doesn’t

• Low volume. A team of 15 asking a few hundred questions a day will almost always pay less for an API than for an idle GPU.
• You need the very best reasoning. For the hardest tasks, the leading hosted models are still generally ahead.
• Nobody will maintain it. A self-hosted model needs updates, monitoring, security patches and capacity planning. Unmaintained infrastructure is a risk of its own.

Private LLM vs API: a practical comparison

The hybrid approach most companies end up with

In practice, the best answer is rarely all-or-nothing. A common pattern:

• Route by sensitivity. Documents classified as confidential go to the self-hosted model; general tasks (marketing drafts, public research) go to a hosted API.
• Redact before sending. Strip names, IDs and account numbers before a prompt leaves your network, and restore them in the answer.
• Keep retrieval private. In a company knowledge base, the document store, embeddings and search index stay on your servers even if the final answer is generated by an API. Only the few relevant passages are sent.

The architecture stays the same either way, so you can start with an API and move the sensitive parts in-house later without rebuilding.

Privacy and compliance: what a private model does and doesn’t solve

Running a model yourself removes a third-party processor, which simplifies a lot of GDPR paperwork. It does not remove your own obligations: lawful basis, access controls, retention limits and the rights of the people whose data you process still apply.

If you fine-tune a model on personal data, be careful. In its Opinion 28/2024, the European Data Protection Board said AI models trained on personal data cannot automatically be considered anonymous; that has to be assessed case by case. For most business use cases, retrieval (RAG) over a controlled document store is safer than fine-tuning on personal data, because you can delete a document and it is gone. If you also fall under the EU AI Act, logging and transparency requirements apply regardless of where the model runs.

A private model also does not protect you from prompt injection. Our guide to AI agent security covers the controls that do.

What a private LLM project involves

1. Data and risk mapping. Which documents, which users, which tasks, and which data must never leave.
2. Model selection and benchmarking. Test two or three open models against your real documents and questions. Public benchmarks are a starting point, not a decision.
3. Infrastructure. GPU servers in your cloud region or on-premise, a serving layer, authentication and network isolation.
4. Retrieval and integrations. Connect the model to your documents and systems, with permissions mirrored from your existing tools.
5. Evaluation and monitoring. Accuracy tests, latency and cost dashboards, and logs for audit.

Frequently asked questions

Are open-weight models good enough for business use?

For summarisation, extraction, classification, drafting and question-answering over documents, modern open models are usually good enough. Test them on your own data before committing.

Do we need to buy GPUs?

Not necessarily. Most teams start with GPU instances in their existing cloud account and only buy hardware when usage is steady and predictable.

Can a private LLM be connected to our CRM or ERP?

Yes. The model is only one part; integrations are built the same way as with hosted models, for example through an MCP server or direct APIs.

How long does it take?

A proof of concept on real documents typically takes a few weeks. A production system with integrations, permissions and monitoring takes longer, depending on scope.

Talk to us about a private LLM for your business

Shkulaku builds AI agents and RAG systems that run on hosted APIs, private open-weight models or a mix of both — chosen by what your data and budget require, not by what is fashionable. Our Enterprise AI plan includes an option for private and local models (see prices). Describe your data and use case, and we will recommend the simplest setup that meets your privacy requirements.