Abigail: A Voice-First AI Operations Platform

• Project Type: Personal / Product in development
• Timeline: Ongoing (phased build)
• Category: AI / Voice / Full-Stack
• Year: 2026

The Challenge

Over the last couple of years I've built the same thing about fifteen times. Jarvis this, Nyaomi that, a voice-chat experiment here, a model council there, a memory daemon, a safety validator for a completely different project. Every one of them started with the same ambition — a proper assistant that lives on my machine, talks back, and gets work done — and every one of them stalled somewhere different. One had great voice but no memory. One remembered everything but couldn't touch a file safely. One could run for days unattended but only knew how to do one job.

The problem wasn't capability. Between all those builds I'd already written every piece I needed. The problem was that the capability was scattered across fifteen repos, half of them coupled to a personal persona, secrets baked into configs, owner face and voice sitting in the source tree, dead experiments cluttering the working code. Nothing was a product. It was a graveyard of good ideas.

Abigail — A Big Artificially Intelligent Loader — is me refusing to build number sixteen. Instead of starting again, the job is consolidation: pick the strongest prior build, strip it of everything personal, and treat every capability I've already proven as an organ to be transplanted into one clean body. The moat here isn't any single trick. It's the assembly. These capabilities took fifteen iterations to build; the value is finally having them in one place.

The Approach

The mental model is anatomical, and I'm leaning into it because it keeps the architecture honest. There's one spine — a kernel and typed event bus that orchestrates everything and handles full-duplex interrupts. Organs plug into the spine. Brain is the tool loop and model routing. Ears, mouth and eyes are voice-in, voice-out and vision. Hands take actions, and danger is the firewall those actions have to pass through. Skills load hot from folders. A Vite dashboard is the face. Each organ owns one job, and the base is the strongest fork I had — the build previously known internally as Jarvis_Final.

Rather than rewrite from a blank page, I mapped out exactly which prior project each future organ gets grafted from. The persistent memory daemon comes from a versioned-identity build I'd already written. The safe-action validator comes from a completely unrelated project — a Minecraft overlord bot whose allowlist-and-clamp pattern generalises perfectly to shell, file and HTTP actions. The unattended jobs engine comes from an AI radio station that had to survive running for days. The self-audit conscience comes from a response-analysis tool. None of these are speculative; they all exist and work in their original homes. The engineering is in de-coupling them from where they were born and making them plug into the same spine.

That's a deliberately different kind of build. It's less "invent" and more "harvest, clean, integrate" — with a hard rule that personal assets, secrets and databases never make it into a shipped build. Phase 0 was the unglamorous part: adopting the base, de-personalising it, renaming the whole config surface from the old persona prefix to a neutral ABIGAIL_ one, and removing the owner's face and voice so what remains is a product, not a diary.

What Was Built

The working anatomy is intact and running today. Concretely, that means:

• Full-duplex, interruptible voice. Wake word, then listen with faster-whisper and Silero voice-activity detection, think, and speak with streaming TTS via Pocket-TTS. Crucially you can cut it off mid-sentence — the interrupt is a first-class event on the spine, not a hack. This is the piece that already works end to end.
• A consent-per-tool brain. The tool loop is built so the model has to consent to every action it takes, routed through a multi-backend model cascade with a tiered context budget. It's not a black box firing commands; every step is an explicit decision.
• A safety firewall on every action. Anything the hands do passes through danger.py. Safe roots are config-driven rather than hard-coded, and raw destructive commands are blocked outright. This is the layer a free chatbot structurally can't offer, because a free chatbot doesn't run on your machine with real access.
• Hot-loadable skills. Capabilities live in skill folders that load without a restart, so extending what Abigail can do doesn't mean redeploying the platform.
• A cockpit dashboard. A Vite front end connected over WebSockets shows status — listening, speaking, idle — plus tool executions, vision results and skill changes, and it accepts typed input as well as voice.
• A test suite kept green. Fourteen test files that have to stay passing through every phase, so consolidation doesn't quietly regress the organs that already work.

The architecture also names the things that make this worth charging for, and is honest about which are live versus grafted later: full-duplex interruptible voice and safe, bounded machine actions exist now; persistent long-term memory, unattended autonomous jobs, a self-audit anti-sycophancy conscience, and an optional multi-model council for hard problems are the phased grafts still ahead.

Where It Stands

I'm going to be straight about status, because that's the whole point of writing these up honestly. Abigail is at Phase 0 — the base has been adopted and de-personalised, the persona coupling is gone, and the working organs (spine, brain, voice, hands, safety, skills, dashboard, tests) are intact and running locally. What works today is the hands-free, interruptible voice loop and safe, firewalled actions on my own machine. That alone is already more useful to me than any of the fifteen builds it replaces, because it's one thing instead of fifteen half-things.

What's ahead is the graft schedule. Phase 1 replaces the weak in-brain memory with a proper durable memory daemon running as its own process, so Abigail remembers across months rather than per-chat. Phase 2 generalises that Minecraft-born validator over the action layer, giving every tool a spec, trust tier and a render/preview/approve flow. Phase 3 turns the survive-for-days radio engine into a general jobs engine for unattended autonomous tasks. Phase 4 extracts the portable self-audit scorer so Abigail can check its own replies for sycophancy and self-correct. Every phase ships behind a gate, and the remaining boxes are watch-and-sign-off items rather than unknowns.

The positioning I'm building toward is a personal AI operations platform sold as a one-time desktop licence, in the region the blueprint targets, plus a founder tier — deliberately not another monthly subscription to a chatbot in someone else's cloud. Whether it ends up a product I sell or a tool I just use every day, the honest version of this case study is the same: I stopped rebuilding, and started assembling. That's the interesting engineering decision, and it's the one I'd make again.

Related Services

Abigail is the deep end of my broader AI Solutions offering. I build custom AI pipelines, local-first assistants, automation workflows and intelligent systems for clients across the UK — voice interfaces, agent tooling, RAG document systems, or something completely bespoke. Take a look at my pricing or get in touch.

If you like grounded, local-first AI, see my DirectoryLM case study — same philosophy, pointed at explainer video instead of voice.

---