# AJRouter: Turning llama.cpp Into a Reusable Local AI Server By AJTheDev — 2026-09-20 Canonical: https://ajthe.dev/case-studies/posts/ajrouter-local-ai-server.html Client: Personal Product · Tech: PowerShell, llama.cpp, audio.cpp, HTML, JavaScript, Windows > Loading a GGUF is the easy part. I built AJRouter because useful local AI needs model routing, predictable startup, resource management, audio services, monitoring, agent integration and a way to install the whole engine again without reconstructing it from memory. --- ## Running a model is not the same as running a service The first version of local AI most people meet is a command: download llama.cpp, point it at a model, open a port. Technically, that is a local LLM server. It is also where the interesting problems begin. A coding agent may need one model for implementation, another for review and a much larger context for reading a repository. A speech workflow needs a different runtime. A GPU-heavy music job cannot simply fight an already loaded language model for the same VRAM. A useful setup has to survive restarts, expose health and logs, preserve configuration, reject unsafe process assumptions and be repeatable on another Windows machine. AJRouter emerged as the control plane around those problems. It does not replace llama.cpp. It manages the processes and configuration around llama.cpp and audio.cpp so the machine behaves like a coherent local AI platform rather than a folder of unrelated executables. ## The machine that forced the right questions The recorded benchmark environment is deliberately unremarkable by AI-server standards: Windows, an Intel i7-9700K, 64 GB of system RAM and an RTX 5060 Ti with 16,311 MiB reported VRAM. Most model storage is on a Samsung 970 NVMe drive, with some larger weights on a SATA SSD. That matters because 16 GB is enough to do serious work, but not enough to ignore architecture. A configuration can load and still be a poor service: it may leave almost no VRAM headroom, spill computation to the CPU, allocate context that the task never uses, evict a better resident model for a tiny request, or spend its whole output allowance reasoning without returning an answer. **“Can this GPU technically load the model?” is a capacity question.** **“Can this machine provide a useful local AI service?” is a systems question.** It includes time to load, prompt processing, decode rate, context fit, output reliability, switching cost, queueing, verification and what else needs the GPU. I have not benchmarked AJRouter on an RTX 3060 12 GB, so I am not going to invent numbers for one. The controls become more important as VRAM gets tighter, but the measurements in this article belong to the 16 GB machine above. ## What the local AI folder became The finished structure separates binaries, configuration and mutable data. Runtime executables live apart from model weights. Logs and process state live apart from configuration. Scripts cover startup, status, tests and benchmarks. AJRouter sits above the runtimes and turns those pieces into managed workloads. AJRouter architecture / control and data planes **Agents & API clients**Codeflow · coding agents · scripts · OpenAI-compatible SDKs **Browser operator**Loopback dashboard · start/stop · configuration · consoles → **llama.cpp text router**OpenAI-compatible API · explicit model profiles · one resident model **AJRouter control plane**Process ownership · GPU handoff · health · logs · settings · updates **audio.cpp workloads**GPU TTS · CPU speech-to-text · GPU music generation **Configuration & mutable storage**Profiles · catalogue · process state · outputs · models · persistent logs Inference traffic goes to the model APIs. AJRouter manages those APIs; it is not pretending to be another inference engine or an invisible proxy. 01 / Text ### llama.cpp router mode An explicit profile catalogue drives automatic model loading behind a local OpenAI-compatible endpoint. The current service limits residency to one model. 02 / Audio ### Separate speech workloads audio.cpp provides a managed GPU text-to-speech service, while Parakeet speech-to-text is configured on CPU. YuE2 music is another managed workload. 03 / Control ### A dashboard that owns processes Start, stop and restart controls are tied to recorded process identity. AJRouter refuses to kill an untracked process merely because it occupies the expected port. 04 / Agents ### A backend tools can actually use Codeflow and other local tools call the same API, select profile aliases and keep their own task state. A GPU-handoff tool coordinates workloads that cannot coexist. ## The useful technical discoveries ### 1. VRAM is a budget, not a yes/no test The text service currently uses one loaded model and one parallel slot. That looks conservative until a profile consumes nearly the entire 16 GB budget. A saved Qwen test at 32K context reported 15,833 MiB GPU use against 16,311 MiB total. The same weights at 24K reported 15,669 MiB with virtually unchanged short-prompt decode speed: 43.08 versus 43.40 tokens per second. The difference was only 164 MiB, but the lesson is larger. Context, KV precision, draft cache and offload settings all spend the same finite budget as model weights. “It loaded” does not tell me how much room remains for the desktop, another workload or a longer real prompt. ### 2. Long context has to earn its allocation The current catalogue defines profiles from 32K to 262K context. Those large profiles exist because I use coding sessions and repository work that can genuinely grow. One saved Gemma 12B long-context test processed a 106,506-token project snapshot, recalled three planted markers and generated at 48.89 tokens per second; the run took 156.1 seconds in total. That is positive evidence that the long-context route can be useful. It is not proof that every fact in a 100K prompt will be recovered, and it is not a reason to allocate 262K for every request. AJRouter exposes context at the profile and workload level because “maximum possible” and “right for this job” are different settings. ### 3. Speculative decoding is a tuning problem The profiles use draft models or model-native MTP where the installed llama.cpp build supports them. The tempting move is to increase the number of speculative tokens and assume more must be faster. The saved calibration showed the opposite. For the standard Gemma 26B profile, MTP4 averaged 108.94 tokens per second over five mixed tasks. MTP6 fell to 88.79; MTP16 fell again to 75.29. In a separate quick comparison, the same 32K target moved from 88.13 tokens per second without a draft to 122.83 with MTP4. Both quick runs stopped at the same 512-token ceiling, so that is decode evidence—not a claim that every end-to-end task became 39% faster. | Saved test | Observed result | What it supports | Important limit | | --- | --- | --- | --- | | Gemma 26B, no draft → MTP4 | 88.13 → 122.83 generation tok/s | A tuned draft helped this quick workload. | Both stopped at 512 tokens; not whole-task speed. | | Gemma 26B MTP calibration | MTP4 108.94; MTP6 88.79; MTP16 75.29 tok/s | More speculative tokens were not better. | Five local tasks on one hardware/configuration set. | | Qwen IQ4, 24K → 32K | 43.40 → 43.08 tok/s; 15,669 → 15,833 MiB | Short-prompt decode stayed flat while context consumed headroom. | Does not measure long-prompt quality. | | Gemma 12B long context | 106,506 prompt tokens; 156.1 s total; 3/3 markers | A specialised long-document route can be practical. | Three markers are not universal retrieval proof. | | Qwen automatic fit/spill configuration | 5.09 generation tok/s | A configuration that loads can still be operationally poor. | Several settings changed, so CPU spill was not isolated. | Measurements were recorded locally on 9 September 2026. They describe these specific GGUFs, prompts, runtime build and settings; they are not universal model or GPU benchmarks. ### 4. Model quality includes finishing the job The benchmark harness did not rank models by tokens per second alone. It checked coding, diagnosis, planning, extraction, tool calls and whether a usable final answer actually appeared. Some reasoning profiles consumed the output allowance internally and returned an empty final. A 12B Q6 test used 11.7 GB of VRAM yet produced worse code than the smaller Q4-family setup. That changed how I think about routing. A fast specialist can be excellent for extraction or formatting. A larger profile may be justified for original code. A thinking profile that notices one extra defect can still be the wrong default if it regularly takes minutes or fails to deliver. Useful infrastructure routes by demonstrated behaviour, not just parameter count. ### 5. Switching models is not free With one-model residency, changing profiles destroys the previous child process and its KV state, then loads the next. Saved switching times on this machine were typically 7–15 seconds. One 17,383-token Qwen prompt took about 21.3 seconds just to process. That makes a supposedly cheap detour expensive. Evicting the current coding model to generate a tiny title can create two model loads and a context rebuild around a trivial request. The better routing decision considers load time, prompt rebuild and execution together. Sometimes the resident model is the cheapest model. ### 6. Workload separation gets more from the same PC AJRouter treats text, speech and music as separate workloads with explicit device assignments. The configured speech-to-text service runs on CPU, leaving the GPU available for a language model. GPU-bound workloads can declare exclusive use. A handoff stops only managed workloads claiming the same physical GPU before starting the next one. The related agent handoff goes further: it records the active model, unloads it, runs the external GPU workflow, verifies requested artifacts and reloads the original alias. Unloading necessarily discards KV cache, and the documentation says so. This is not magic concurrency; it is controlled resource ownership. ### 7. An OpenAI-compatible endpoint is the integration layer The llama.cpp router exposes the familiar `/v1/chat/completions` shape. The repository's PowerShell tests, benchmark harnesses, state-machine client and Codeflow all use it. That means the local server can sit behind tools written for an OpenAI-compatible API without each client learning how the model process is launched. ``` $body = @{ model = 'your-profile-alias' messages = @(@{ role = 'user'; content = 'Review this function.' }) temperature = 0 max_tokens = 1024 } | ConvertTo-Json -Depth 10 Invoke-RestMethod ` -Uri 'http://127.0.0.1:8080/v1/chat/completions' ` -Method Post -ContentType 'application/json' -Body $body ``` The dashboard itself binds to `127.0.0.1` by default. Managed APIs retain their configured bind address, so exposing an API beyond the machine remains an explicit configuration decision rather than something the article quietly assumes. ### 8. Operations are part of model performance AJRouter shows GPU utilisation, VRAM, temperature, process ID, port ownership, health and the resolved native command. Each workload has a persistent console backed by stdout and stderr files. Closing the dashboard leaves managed APIs running; stopping everything is a separate deliberate action. Configuration edits are constrained to known fields. A model-profile edit creates a timestamped backup before changing `models.ini`. Runtime update controls can check and install the configured Windows CUDA builds, while a dirty development checkout is left alone. None of that improves a benchmark screenshot, but all of it improves whether I can depend on the server tomorrow. ## The configuration is where the platform becomes specific AJRouter does not scan a model folder and pretend every file is interchangeable. The text router receives an explicit profile file. Global defaults set the baseline; individual profiles override context, offload, template and draft behaviour. ``` [*] parallel = 1 flash-attn = on cache-type-k = q4_0 cache-type-v = q4_0 cache-type-k-draft = q4_0 cache-type-v-draft = q4_0 n-gpu-layers = all [example-32k] ctx-size = 32768 spec-type = draft-mtp spec-draft-n-max = 4 ``` That excerpt is deliberately sanitised: no private filesystem paths and no claim that those values are universal defaults. It shows the decisions AJRouter makes manageable—parallelism, Flash Attention, KV precision, GPU offload, context and draft depth. A different GPU or model family can require a different answer. ## From personal stack to a distributable engine The point at which this stopped being “my setup” was the installer. The repository now has one packaging command: ``` .\install.ps1 -CreatePackage ``` It validates required executables, builds a dated `AJRouter-engine-.zip` under `dist`, adds a manifest and reports the uncompressed payload. The current neutral distribution validates at 107 payload files and 1.77 GB before compression. Engine package flow **Build**Run `-CreatePackage` against the verified source tree. **Send**Distribute the engine ZIP—not model weights or mutable runtime data. **Extract**The recipient unpacks it to a normal Windows folder. **Install**The script provisions the engine, command shim and optional shortcuts. Models are installed separately. That keeps the engine package independent of model licensing, size and each user's hardware choices. After extraction, the recipient runs: ``` powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` By default the installer copies AJRouter, the included llama.cpp and audio.cpp runtimes, dashboard resources, neutral configuration, scripts and documentation into `%LOCALAPPDATA%\AJRouter`. It creates an `ajrouter` command on the user PATH plus Desktop and Start Menu shortcuts. The destination, PATH change and shortcuts are all optional parameters rather than hard-coded requirements. On a clean install, `-ApiHost Auto` resolves the recipient computer's current LAN IPv4 address for workload APIs; `-ApiHost Localhost` keeps them on `127.0.0.1`. The dashboard itself remains loopback-only, and the installer never opens or modifies Windows Firewall rules. A repair install preserves the existing `config` directory when an AJRouter configuration is already present. `-ForceConfig` deliberately replaces it. Installation also creates fresh folders for logs, models, outputs, samples, staging, process state and verification data. The package does **not** include the model-storage tree, saved model selections, model download recommendations, personal voice samples, logs, outputs or AJRouter runtime state. Its file filter also excludes model-weight extensions, build intermediates, `.env` files, dependency caches and private benchmark or integration material. I still treat packaging as a release process rather than a secret scanner: a controlled exclusion list is not a substitute for reviewing the final archive. I ran the installer's validation path under both Windows PowerShell 5.1 and PowerShell 7, and both reported the same valid 1.77 GB payload. I also built and inspected the ZIP, performed a clean extracted install with automatic LAN detection, confirmed repair-install configuration preservation and tested an explicit localhost reset. These checks are stronger than parser validation, while still narrower than claiming broad clean-machine compatibility. ## Who AJRouter is for AJRouter is for people who have moved past “can I run a model?” and now need a local AI server they can repeatedly use: - **Local AI developers** who want explicit model profiles and visible native commands instead of opaque defaults. - **Coding-agent users** who need a stable OpenAI-compatible local API, long-context options and measured task routing. - **llama.cpp users on Windows** who want process management, logs, monitoring and repair installs around the runtime. - **Multi-model experimenters** who care about residency, switching cost, KV cache choices and speculative-decoding calibration. - **Voice and agent builders** who want text, TTS, ASR and GPU-heavy generation treated as coordinated workloads. - **Consumer-GPU owners** trying to extract more useful work from finite hardware without pretending software removes its limits. If all you want is one model in one terminal, llama.cpp already does that job brilliantly. AJRouter is for the stage after that. ## AJRouter licences The product is offered in two editions. The engine is the same practical infrastructure described above; the intended use is what changes. Personal ### AJRouter Personal £39.99 personal licence For personal and non-commercial local AI use. [Buy Personal on Gumroad →](https://ajthedev.gumroad.com/l/wvdeij?variant=Personal) Commercial ### AJRouter Commercial £79.99 commercial licence For businesses, revenue-generating projects and paid client work. [Buy Commercial on Gumroad →](https://ajthedev.gumroad.com/l/wvdeij?variant=Commercial) Need help deciding which applies, or want to confirm the current licence terms before buying? [Contact me about AJRouter](https://ajthe.dev/contact/). If you need a bespoke local-AI deployment rather than the packaged engine, see my [AI solutions](https://ajthe.dev/ai-solutions/). ## Questions people ask about a local AI server Is AJRouter another model or a replacement for llama.cpp? No. AJRouter is the control and deployment layer around the included llama.cpp and audio.cpp runtimes. The model APIs remain the data plane. Does the AJRouter package include AI model weights? No. The engine package deliberately excludes model weights. You add models separately, which keeps model licensing and hardware choices with the user. Was AJRouter benchmarked on an RTX 3060 12 GB? Not for the results on this page. The evidence here comes from an RTX 5060 Ti with 16 GB of VRAM. A 12 GB VRAM local LLM setup makes context, quantisation and offload choices even more consequential, but I will not publish RTX 3060 performance numbers I have not measured. Does AJRouter remove the hardware limits of local AI? No. It exposes and manages the choices that determine how effectively the available CPU, RAM and VRAM are used. It cannot make those physical limits disappear. Can existing tools call the local API? The text service exposes an OpenAI-compatible endpoint, and the repository already uses it from PowerShell, Python clients, Codeflow and coding-agent integrations. Client compatibility still depends on the exact API features that client expects. ## The bigger idea Local AI conversations often end at the GPU shopping list. More VRAM genuinely opens options, and software does not repeal that fact. But building AJRouter pushed me towards the opposite question: **how much more can I make the hardware I already own do?** The answer was not one magic quant or one spectacular benchmark. It was a series of less glamorous decisions: choose the profile for the task, allocate context intentionally, measure the draft depth, keep the right model resident, move speech recognition to CPU when that separation helps, expose a standard API, preserve configuration and make the engine installable. Raw inference is the motor. The routing, resource policy, monitoring, clients, persistence and deployment around it are what turn that motor into a vehicle. AJRouter is the system I built because I wanted to drive it every day. --- ## Want the working engine instead of rebuilding the stack? AJRouter packages the control plane, runtimes and Windows deployment workflow described here. For custom local AI, agents or audio infrastructure, I can build around your actual workload. [Get AJRouter on Gumroad →](https://ajthedev.gumroad.com/l/wvdeij) [Ask about AJRouter](https://ajthe.dev/contact/) before buying if you are unsure which licence applies.