I have just finished Phase 4 of my vLLM-on-Kubernetes project: a tiny model, a Helm chart, a validation ladder run from my laptop over WireGuard. The pod was Ready. The deployment had rolled out successfully. The health endpoint returned two hundred, over and over, hundreds of times in the logs.

Inference still took eight minutes.

Not eight minutes to deploy. Eight minutes for a single chat completion — the kind of request that would be one curl in a normal API. The platform looked fine. The workload was not.

That gap is what stuck with me. Not the Helm values, not the Graviton dtype choice, not the service-link footgun that crashed vLLM on startup until I disabled Kubernetes’ default env injection. Those belong in the repo. This is the judgment call: when cold start costs minutes, green probes stop meaning what you think they mean — and running inference on a homelab CPU made that gap impossible to ignore.

This is not really “ML vs web”

I want to be honest about the framing, because an earlier draft of this post oversold it.

A lot of what I saw was CPU slowness, not some special failure mode of machine learning on Kubernetes. vLLM on a small Graviton node compiles, loads weights, and generates the first token on hardware with no GPU. Eight minutes is physics plus software, not Kubernetes refusing to understand transformers.

Web apps can have the same shape of problem. A JVM warming up. A cache still filling. A database pool not ready. A naive /health that returns two hundred while the app cannot yet serve real traffic. Kubernetes does not know your product’s definition of ready. It knows what your probe returns. vLLM’s health endpoint answering before inference is ready is the same class of mistake as a Spring actuator that lies — ML just stretched the gap long enough that I could not pretend the probe was enough.

If I rerun Phase 4 on GPU in Phase 5, I expect the numbers to shrink. The lesson might feel less dramatic. The underlying issue would remain: cheap health signal, expensive readiness work.

Health checks answer a narrow question

Kubernetes asks: is the process up and responding on a port?

For workloads with short startup, that is often close enough. For vLLM on CPU, the process can listen long before the model is compiled, weights are loaded, and the engine can actually tokenize your prompt. The probe sees HTTP. It does not see inference.

So you get a pod in Running, Ready one of one, restart count zero — and a functional test that sits there for six, seven, eight minutes while the engine does work that never shows up in a liveness probe. The logs fill with health checks. It looks alive. It is busy. It is not serving in any sense a client would recognize.

I knew CPU inference would be slow. I did not expect how misleading the platform signals would feel while that slowness played out — not because K8s is broken for ML, but because I was still reading them with a web-app mental model.

The scary logs were mostly noise

Mid-wait, vLLM logged shared-memory broadcast warnings — no block available for sixty seconds, typically when processes hang or compile. In a web app that would send me to incident mode.

Here it was the engine chewing through a first request on a small Graviton node. Not a crash. Not OOM. Just CPU doing CPU things while my check script counted seconds and told me to watch the pod logs if I got impatient.

That part is somewhat runtime-specific: inference engines log in ways that look like hangs when they are doing compilation or quantization. But the operating mistake is universal — interpreting slow progress as failure because your telemetry was shaped for fast request/response services.

My real “ready” signal was inference, not Ready

I have written before about done when the script says so. Phase 4 sharpened that for me.

Gate three — deploy, pod Ready — passed in minutes. Gate four — in-cluster chat completion with a parsed JSON response — passed after hundreds of seconds. That was the first moment I believed the cluster could run the product, not just host a container.

Gate five almost failed for a reason that had nothing to do with Kubernetes or ML: it looked for startup lines in the last two hundred log lines. By then the buffer was nothing but health checks. The model had started hours earlier. The check was wrong. Even my corrective automation had assumed log shapes from a workload that starts in seconds, not minutes.

So now I hold two ideas at once:

Platform ready — scheduled, probed, not restarting.

Workload ready — a real request succeeded end-to-end, with output you can validate.

For Phase 4 I will not collapse them. Helm --wait and Ready conditions get me to the first. Only a functional gate gets me to the second. That split is not unique to inference — any workload with a long warm-up needs it. I just happened to learn it on a CPU-bound LLM where the warm-up took longer than my patience.

What I think this means

If you deploy a workload with minute-scale cold start without changing how you define success, you will ship false confidence. Dashboards will look fine. On-call will wonder why “the pod is up” but the first real request takes forever. Someone will widen timeouts until the pain stops showing up in alerts without fixing the contract.

That is not a Kubernetes bug. Kubernetes is doing what it was designed for: keep processes alive and route traffic to endpoints that answer cheap probes. Your job is to define ready in terms of the work, not the port — and to accept that on CPU inference, the gap between those two can be measured in minutes.

I am not arguing against K8s for ML. I am arguing against importing default operating instincts unchanged: probe passed, therefore ship it. You need a done-when that matches the workload — inference succeeded, not just port open — and you need humans who know eight minutes can be success on a homelab CPU while every health check says everything is fine.

Phase 5 is GPU. I expect the gap to narrow. I do not expect probes to suddenly encode “model ready” without me asking them to. I will keep the ladder. I will keep gate four harder than gate three.

The step-by-step and the exact outputs live in the repo. Phase 4 is marked complete because inference returned JSON — not because the pod looked happy in kubectl get.