TL;DR: Self-hosting an LLM comes down to three options: subscribe, buy, or rent and build. Renting dedicated hardware gives you the ownership buying promises, without the procurement, and none of the per-token exposure of subscribing. This guide covers all three, then walks through exactly how to build the third, step by step.
When someone asks which team is using which model, on what data, and why last month’s compute bill jumped, “we don’t track it that closely” doesn’t hold up for long. That question is usually what puts self-hosting an LLM on the table. Not because it’s cheaper on paper. Because it’s the option where you can answer it. This guide is for whoever ends up building that answer: the IT manager standing up the stack, or the startup CTO who is also the ops team.
Running a language model comes down to three options: subscribe, buy, or rent and build. You can pay a provider per token, or purchase the GPUs and run everything in-house. Or rent dedicated hardware and build the stack yourself. Which one fits depends on the company, the workload, and how predictable the usage is. This guide covers the trade-offs of the first two, then spends the rest of its time on the third: self-hosting an LLM, step by step.
The two conventional paths
Subscribe: pay a provider per token
Every prompt, every uploaded document, every conversation goes to a provider you don’t control. They set the price, and they can change it. Rate limits work the same way. Even the model itself can change underneath you, altering behavior you’ve already tuned a prompt against in production. None of this is a criticism of any single provider. It’s what happens by default when the model isn’t yours. For plenty of teams it’s the right call: no procurement, no ops burden, someone else on call.
Buy: purchase the hardware
A team decides to take control, so they procure GPUs, rack them, install drivers, and inherit every 2 a.m. incident that comes with owning physical infrastructure. The upside is real: no per-token bill, no provider changing terms underneath you. The trade-off is that a small team now runs a data center as a side project. That’s on top of the actual work they were hired to do. For a company with the ops capacity to carry that, it’s a defensible choice.
Most companies default to one of these two. Self-hosting an LLM, the third path, rarely gets explained.
The third option: rent the hardware, build the stack yourself
Self-hosting an LLM doesn’t require owning a data center. It requires a GPU you control, and you can rent that GPU the same way you rent any other server. The distinction that matters is between renting compute from a provider (bare metal, no hypervisor, the whole card) and renting a model from a provider (an API call that returns text). Renting compute gives you everything buying promises, minus the procurement and the racking. Most people mean the second one, renting a model, when they say “using AI,” and it’s a different thing entirely.
On rented bare metal, the weights are a file you hold. Nobody can withdraw the model you built on, change the acceptable-use policy underneath you, or update the model’s behavior while your tuned prompts are live in production. The prompts, documents, and conversation history stay on a server you control. No model provider, and no jurisdiction you didn’t choose, sits anywhere in the request path. And because the hardware is rented rather than owned, the expense is a fixed monthly line item. It doesn’t start depreciating the day it arrives, the way a purchased asset does.
This isn’t the right call for every workload. If usage is occasional or spiky, a per-token API is cheaper, and someone else carries the operational weight. Self-hosting an LLM pays off under sustained, predictable usage, the kind where you can already see the API bill climbing every month. If that’s not your situation yet, this guide is still worth reading, because it tells you what to watch for.
What you’re building
Self-hosting an LLM breaks down into three separate pieces, and understanding that they’re separate is what makes the rest of this guide make sense.
The model. A file, or set of files, containing the trained weights.
The inference server. The software that loads the model onto the GPU and answers requests. This is where the real engineering decisions live.
The interface. What a person or application talks to. A chat window, or an API endpoint called by another program.
You can swap any one of these three without touching the other two. That’s the whole appeal: you’re not locking yourself into one vendor’s stack. You’re assembling a system out of interchangeable parts.
Self-hosting an LLM, step by step
Sizing the hardware and the model
- Start with the GPU, not the model. VRAM is the hard limit on what you can run, and unlike disk or system memory, you can’t extend it after the fact. A single card keeps the setup simple. It’s enough for a capable assistant: drafting, summarizing, answering questions about your own documents, extracting structure from messy text, reviewing code. It won’t match the largest commercial models on the hardest reasoning tasks, and it doesn’t need to for most of what teams use these tools for day to day.
- Pick the model, and check the license before you get attached to it. Parameter count (8B, 32B, 70B) drives quality and memory footprint, and the relationship isn’t linear, so more isn’t automatically better for your use case. Make sure you’re grabbing the instruct or chat variant, not the base model. A base model just continues text; it will look broken in a chat interface when it isn’t. Read the license on the model card. “Open weights” doesn’t always mean “use for anything.” Some carry commercial-use conditions worth a five-minute check before you build around them.
Precision and the VRAM math
- Decide how much precision you need. Full 16-bit precision is the baseline. 8-bit cuts memory roughly in half and is hardware-accelerated on recent GPUs, which makes it a reasonable default rather than a compromise. 4-bit cuts it to roughly a quarter, with a real but often acceptable quality cost. A larger model quantized down usually beats a smaller model at full precision, up to a point.
- Do the VRAM math before you commit to a model. Three things compete for the same memory: the weights (parameter count times bytes per parameter), the KV cache (which grows with conversation length and with how many conversations are running at once), and headroom (leave 10 to 20 percent unallocated). The KV cache is the number teams forget. It’s the reason a model that looks like it fits will still choke under three concurrent users. Long context for one person, or shorter context for many people. That trade-off shapes everything downstream.
If you need a number for the budget conversation: cloud GPU spend crossing roughly $25,000 to $40,000 a month, or utilization sustained above 40 to 70 percent, is the documented point where dedicated hardware starts beating on-demand rental economics. Below that, you’re paying to keep hardware idle most of the month.
Software: the server and the interface
- Choose the inference server based on who’s depending on it. Ollama or llama.cpp are fine for confirming the GPU works and getting a first answer out of the machine. Neither one was built to scale well with multiple simultaneous users. Both can run parallel requests, but neither is optimized for it. vLLM is the one to run in production: continuous batching keeps the GPU busy across overlapping requests instead of finishing one before starting the next. PagedAttention manages the KV cache in small blocks instead of reserving a fixed chunk per conversation, which means more simultaneous users fit in the same VRAM. All three expose an OpenAI-compatible API, so an application already built against a commercial provider works after a URL change and nothing else.
- Put an interface on it, or don’t. Open WebUI gets you a familiar chat experience, accounts, and conversation history fast. LibreChat is the answer when single sign-on against an existing directory, or centrally disabling features, matters more than speed of deployment. If the goal is adding a capability to an existing application, skip the interface entirely and call the API directly.
Whichever you choose, keep the API bound to localhost until you deliberately place it behind a reviewed proxy. An unprotected inference endpoint is free compute for anyone who finds it, and automated scanners look for exactly that. Loop security in on this step early, not after something is already live.
Keeping it running
- Watch two things, not one. Hardware metrics tell you if the card is busy. Inference metrics tell you if people are waiting. A GPU sitting at 100 percent utilization is either completely healthy or badly overloaded, and hardware metrics alone can’t tell you which. Alerting on GPU utilization by itself is the most common monitoring mistake teams make here.
Add a third thing to the list once the endpoint is live: prompt injection and system-prompt extraction. It’s an emerging risk specific to self-hosted stacks, and it tends to land on whoever runs the infrastructure, even though it’s a security problem, not an ops one. Settle that ownership question before launch, not after an incident forces the conversation.
Where this breaks down
The honest limits
Operational responsibility moves to you, and it’s more than the usual disclaimer suggests. Teams running this in production report 4 to 8 hours a month on patching and monitoring alone, before counting a single incident response. Get the VRAM sizing wrong and the failure isn’t loud: weights spill into system memory, the model doesn’t crash, it just gets 50 to 100 times slower. That shows up as an angry user before it shows up on a dashboard. Model swaps carry their own cold-start latency, so the first request after a change is never the fast one.
A model that fits on a single GPU is not a frontier model. That’s a single-GPU VRAM limit, not a limit on self-hosting itself. Move to a multi-GPU box, up to 8 GPUs on Leaseweb’s bare metal, and the same rented-hardware approach scales to larger, more capable models through tensor parallelism. This guide covers the single-GPU build because it’s the simplest starting point, not because it’s the ceiling. Self-hosting rewards steady usage. If your traffic is unpredictable or occasional, you’re paying for idle capacity that a per-token API wouldn’t charge you for.
For startups specifically
If you’re at a startup rather than an enterprise, self-hosting an LLM still works. Just run the same math with sharper numbers. Self-hosting typically runs 3 to 5 times the raw GPU price once DevOps overhead, monitoring, and downtime are included. The break-even point sits around 2 to 5 million tokens a day, depending on the workload. Know that number before you commit engineering time to it. Below it, a per-token API is still doing you a favor. Above it, the math in this guide starts working in your direction.
Our point of view
Buying and subscribing are both defensible choices, and which one fits depends on the company and the workload. Where we’d push back is on treating those two as the only choices. Renting dedicated bare metal gets you the ownership that buying promises. It gets you the operational simplicity of not owning physical hardware. And it gets you none of the per-token exposure of subscribing.
Most teams haven’t heard this framing because most providers only sell one side of it: hyperscalers sell the API, and colocation providers sell the rack. Almost nobody sells a GPU, or a set of them, you fully control on a monthly bill with none of the procurement.
That’s the gap worth naming: the option where you can answer where the data went, and what drove the bill. Self-hosting an LLM this way is the one we build for.
Where to start
If you’re already watching a per-token bill climb every month, or you’ve hit a point where you’d rather know which jurisdiction can ask for your data than find out after the fact, the fastest way to find out if this is worth it is to run the VRAM math against your model and your usage. If you want a second pair of eyes on that math, on the ops load, or on a bare metal GPU to test it on, that’s a conversation we’re glad to have. Self-hosting an LLM starts with that one number.