Llama CPP Server Manager

A lightweight wrapper around llama.cpp's llama-server that simplifies installation, configuration, and lifecycle management of a local LLM inference server. It supports OpenAI-compatible REST API endpoints, making it easy to drop into existing tooling and workflows.

---

Table of Contents

• Requirements
• Installation
• Install Llama CPP
• Configuration
• Usage
• Model Management
• Starting the Server
• Stopping the Server
• API Usage

---

Requirements

• Python 3.12.3+ with pip — Download Python (pip is included with Python 3.4+)
• macOS, Linux, or Windows (WSL)
• A Hugging Face account (for downloading models)

---

Installation

Create and activate a Python virtual environment before installing dependencies.

Create the environment:

# macOS / Linux / WSL
python3 -m venv .venv

# Windows (native)
python -m venv .venv

Activate the environment:

# macOS / Linux / WSL
source .venv/bin/activate

# Windows (native)
.venv\Scripts\activate

Install the project dependencies:

pip install -r requirements.txt

Install the Hugging Face CLI for model management:

pip install hf-cli

---

Install Llama CPP

Run the install command to download and set up llama.cpp:

./llama-server-manager --install-llama

This will walk you through an interactive menu to install llama.cpp. The script attempts to detect your OS and hardware automatically — review each option carefully to make sure the correct build is selected for your system.

Once installation completes, a llama-cpp/ folder will appear in your install directory and you're ready to run the server.

---

Configuration

On first run, the wrapper generates a conf.json with safe defaults. You can customize it to pass additional options directly to llama-server.

Example conf.json:

{
  "options": {},
  "llama-server": {
    "options": {
      "host": "0.0.0.0",
      "port": "11235",
      "models-max": "1",
      "sleep-idle-seconds": 600
    }
  },
  "logging": {
    "enabled": true,
    "level": "INFO",
    "file": null
  }
}

Key options:

Option	Default	Description
host	127.0.0.1	Set to 0.0.0.0 to expose the server on your local network
port	8080	Change if another process is already using port 8080
models-max	1	Maximum number of models loaded simultaneously — keep at 1 if VRAM is limited
sleep-idle-seconds	600	Unloads the model after this many seconds of inactivity (similar to Ollama's behavior)

---

Usage

Command Reference

Command	Description
./llama-server-manager	Start the server
./llama-server-manager [llama-server args]	Start the server and pass arguments directly to llama-server
./llama-server-manager --install-llama	Download and install the latest llama.cpp release
./llama-server-manager --update-llama	Update an existing llama.cpp installation to the latest release
./llama-server-manager --self-update	Pull the latest manager code from GitHub and restart
./llama-server-manager --stop-server	Gracefully stop a running llama-server

Command Details

[llama-server args] — Any additional arguments are passed through directly to llama-server, one at a time. Refer to the llama.cpp server documentation for the full list of supported arguments.

./llama-server-manager --some-llama-arg value

--install-llama — Run this once after cloning the repo to download and install llama.cpp. The installer will attempt to detect your OS and hardware, but review each prompt carefully to confirm the correct build for your system.

./llama-server-manager --install-llama

--update-llama — Updates your existing llama.cpp installation to the latest release without needing to reinstall the manager or reconfigure anything.

./llama-server-manager --update-llama

--self-update — Pulls the latest version of the manager itself from GitHub and restarts. No prerequisites required.

./llama-server-manager --self-update

--stop-server — Gracefully stops a running llama-server process.

./llama-server-manager --stop-server

---

Model Management

Note: Built-in model management commands are coming soon to the wrapper.

In the meantime, there are two ways to download models:

Option 1 — Hugging Face CLI:

hf download {model-name}

Option 2 — llama-cli (downloads directly into llama.cpp's format):

llama-cli -hf {model-name}

Both methods work well. Use llama-cli if you want the model pulled and placed directly in a format ready for llama-server.

---

Starting the Server

./llama-server-manager

This starts llama-server as a background process. Output is streamed to your terminal, but you can safely close the terminal window — the server will continue running.

---

Stopping the Server

./llama-server-manager --stop-server

This cleanly stops the llama-server process.

---

API Usage

llama-server exposes an OpenAI-compatible REST API, so you can use it as a drop-in replacement with any OpenAI SDK or client.

Chat Completions:

curl http://localhost:11235/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-name",
    "messages": [
      { "role": "user", "content": "Hello!" }
    ]
  }'

Using the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11235/v1",
    api_key="not-needed"  # llama-server does not require an API key
)

response = client.chat.completions.create(
    model="your-model-name",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

Update base_url to match the host and port values in your conf.json.

---

Read My Article About This Journey

I Built a Local AI Coding Assistant on Consumer Hardware…and It Works. I think.