How to monitor a llama.cpp or vLLM server (and get an alert when it goes down)

llama.cpp's llama-server, vLLM, LM Studio and most other local inference servers speak the OpenAI API. Here's how to check they're healthy by hand, and how to get a Discord, Slack, Teams or Telegram alert when one dies or slows down, without opening a port.

Check by hand

Both servers have a health endpoint and the OpenAI model list:

# llama.cpp (llama-server, default port 8080)
curl -i http://127.0.0.1:8080/health      # 200 when ready; 503 while the model is still loading
curl -s http://127.0.0.1:8080/v1/models

# vLLM (default port 8000)
curl -i http://127.0.0.1:8000/health      # 200 when the engine is healthy
curl -s http://127.0.0.1:8000/v1/models

Both can also expose Prometheus metrics at /metrics (vLLM by default, llama.cpp with --metrics). If you already run Prometheus and Alertmanager, scrape those. If you don't, read on.

What usually goes wrong

The process dies, often running out of memory on a long context or a big batch, and nothing restarts it.
The box reboots after an update and the server isn't set up as a service.
It's up but slow: the model didn't fit on the GPU, another job is using the GPU, or a driver update broke CUDA, so generation crawls.

Get an alert with Paw Pulse

Paw Pulse is a free, push-based monitor for home LLM servers. A small sh script on the box asks your server for /v1/models and pushes a heartbeat out. If the server stops answering, or the heartbeats stop altogether (the box is off), you get one alert, and one more when it recovers.

Create a monitor and install the client. The steps are the same as in the Ollama guide (no sign-up, about five minutes).

Point the client at your server:

# llama.cpp
LLM_KIND=openai LLM_BASE_URL=http://127.0.0.1:8080
# vLLM
LLM_KIND=openai LLM_BASE_URL=http://127.0.0.1:8000
# LM Studio's local server
LLM_KIND=openai LLM_BASE_URL=http://127.0.0.1:1234

Started the server with --api-key? Add LLM_API_KEY=<key>. The key is only used for requests to your local server and is never sent to Paw Pulse.

A full cron line looks like:

*/5 * * * * PAWPULSE_ID=<id> PAWPULSE_TOKEN=<ingest_token> LLM_KIND=openai LLM_BASE_URL=http://127.0.0.1:8080 /usr/local/bin/pawpulse.sh

Alert when it's slow, too

Add MEASURE_TPS=1 MEASURE_MODEL=<model id from /v1/models> and the client times a 16-token completion on each beat. Then set a floor on the monitor:

curl -X PATCH https://pulse.dougalpaw.com/v1/monitors/<id> \\
  -H 'authorization: Bearer <manage_token>' -H 'content-type: application/json' \\
  -d '{"slow_tps_threshold":10}'

OpenAI-compatible servers don't report generation timing the way Ollama does, so the number is completion tokens over wall-clock time. It includes prompt processing and network overhead, so it reads a little low; it's a health signal, not a benchmark. Pick a threshold well below your normal speed.

Things to know

The GPU-offload check (min_gpu_pct) is Ollama-only for now, because it reads Ollama's /api/ps.
Run the client on the same box as the server, so "box is off" and "server is down" both get caught.
Paw Pulse is open source (MIT), needs no account and never sends your prompts, hostname or IP. Read the client script before you run it.