# Lyric Video Builder: Frame-Perfect Karaoke Videos in the Browser By AJTheDev — 2026-07-10 Canonical: https://ajthe.dev/case-studies/posts/lyric-video-builder.html --- **Category:** AI & Automation **Client:** Personal Project / Product **Duration:** Ongoing **Tech:** Python, Flask, PyTorch, faster-whisper, WhisperX, Demucs, librosa, Playwright, WebCodecs, FFmpeg Upload an MP3, get a karaoke-style lyric video where every word lands exactly when it's sung. GPU audio analysis, Whisper transcription with forced alignment, and in-browser WebCodecs rendering — no timeline, no manual keyframing, no drift. - **Project Type:** Personal / Product - **Timeline:** Ongoing - **Category:** AI / Audio / Video - **Year:** 2026 ## The Challenge Making a lyric video the normal way is a slog. You drop the track into an editor, then you sit there scrubbing back and forth, nudging each line of text so it appears bang on the beat and disappears when the singer moves on. For a three-minute song that's an afternoon of tedious keyframing, and the moment you want a second song you start again from scratch. It doesn't scale, and honestly it's the least creative part of making anything. The two things that make a lyric video feel *right* are timing and musicality. Timing means the words have to hit exactly when they're sung — not roughly, exactly, because the human eye is brutal at spotting text that's even a fraction of a second out of sync with a voice. Musicality means the visuals should feel like they belong to the track — pulsing on the beat, reacting to the bass and the hats — rather than sitting there as static captions over a song. So the goal was a tool where you upload an MP3, pick a look, and get a finished, frame-perfect karaoke-style video back with the alignment done for you. No timeline. No manual timing. This one also feeds directly into my own [Karaoke video product on Gumroad](https://ajthedev.gumroad.com/l/karaokevids) — being able to turn a song into a synced lyric video quickly is the whole point. ## The Approach The trick to killing the manual timing is to *measure* it instead of guessing it. Everything the tool does keys off two things it extracts from the audio itself: where every word is sung, and where every beat lands. For the words, it runs Whisper transcription and then, crucially, **forced alignment** on top. The transcription (using distil-large-v3, an English model that's near-large accuracy but far faster on GPU) tells you *what* is sung; WhisperX forced alignment with wav2vec2 then pins *when* each individual word starts and ends down to word level. Before any of that, it can optionally run Demucs vocal separation to pull the vocals out of the mix, which gives the transcriber a much cleaner signal to work from on a busy track. If WhisperX isn't available it falls back to faster-whisper's own timestamps — less precise, but it still produces a video. And if you already have the exact lyrics, you can supply them and they get fuzzy-matched onto the detected timings or force-aligned as ground truth. For the musicality, librosa does per-frame beat tracking and BPM detection, plus a multi-band frequency analysis splitting the track into bass (the kicks), highs (hats and sibilance) and overall intensity. Those envelopes become the video's reactivity: word entrances get snapped to the nearest beat so pop-ins feel deliberate rather than floaty, glow depth follows the bass, particles follow the highs, and the beat pulses ride on the animation. Words held on a long note get auto-flagged for emphasis and rendered larger with extra glow. The design principle underneath all of it is determinism. The renderer draws each frame as a pure function of frame time against the word timings — `drawFrame(hue, reactivity, t, {bass, high, intensity, beats})` — so synchronisation isn't something you tune, it's a property of the maths. Get the timings right once and every frame is exactly where it should be. ## What Was Built The main way in is a local Flask web app. You upload an audio file, pick resolution, FPS and a colour style, choose a lyric animation style, optionally paste in your own lyrics and toggle the background, then watch a live progress log as the pipeline runs: audio analysis into a beat grid and band envelopes, optional vocal separation, Whisper transcription plus forced alignment, then frame rendering and audio mux. There's also a direct Python API for scripting or batch work, driving the same audio analyser, lyrics engine and renderer. There are five lyric animation styles, all frame-perfect: **Karaoke** (the default — a full line with a character-level wipe as each word is sung), **Word Pop** (words bounce in with an overshoot at their exact start time), **One Word** (a single bold word centre-screen with an impact zoom), **Slide Up** (words rise in and drift away on exit) and **Typewriter** (letters type out across each word's sung duration). On top of that sit seven colour styles — rainbow, neon, fire, ocean, sunset, cyberpunk and pastel — and a signature rainbow HSL outline with an animated gradient glow that I've deliberately kept untouched because it's the look. Backgrounds are either a classic dark gradient or an audio-reactive WebGL aurora shader that automatically falls back to classic if WebGL isn't available. Output resolutions run from 400×712 all the way up to 2160×3840, so 9:16 portrait for Shorts through to 4K. The rendering is the part I'm most pleased with technically. Frames are drawn in a headless Chromium canvas with GPU flags and encoded **in the browser** via WebCodecs — the frames never leave the page as images — then ffmpeg stream-copies the raw H.264 into an MP4 with the audio muxed back in. Because encoding happens in-page and streams to disk, it's memory-efficient and quick: the README clocks it at roughly 1.65× realtime, so a three-minute song renders in about two minutes including transcription, and a typical end-to-end run from upload to finished video is around four minutes. GPU acceleration carries Whisper, Demucs and the Chromium rasterisation; on CPU it works but it's slow, which is expected. ## Where It Stands This one's a working tool, not a concept. The full pipeline runs — upload to finished MP4 — with a web GUI, a scripting API, and a regression test plus web-layer and end-to-end tests covering the pipeline. The five animation styles, seven colour palettes, both backgrounds and the resolution range from portrait to 4K are all in. It leans on a proper stack of open models — distil-large-v3 for ASR, WhisperX for alignment, Demucs for separation, librosa for the musical analysis — glued together so the person using it never has to think about any of that; they upload a song and get a video. I'm honest about the rough edges the README calls out: first run downloads a couple of gigabytes of models, the aurora background needs WebGL or it quietly falls back, and CPU-only rendering is slow enough that you really want CUDA. The roadmap I'm eyeing next is the stuff that turns it from "render and done" into a proper editor — a live preview with scrubbing, a timing editor to drag word boundaries when the alignment gets a tricky lyric slightly wrong, AI-generated backdrops per section, and batch rendering a queue of songs. The reason it exists is simple: I make karaoke and lyric videos, and I refuse to spend an afternoon keyframing text every time. If you want the finished article rather than the build, my [Karaoke video product is on Gumroad](https://ajthedev.gumroad.com/l/karaokevids). And if you've got an audio-to-video or media-automation problem of your own, that's squarely the kind of thing I build. ## Related Services This sits inside my broader [AI Solutions offering](https://ajthe.dev/ai-solutions/) — audio and video automation, transcription and alignment pipelines, and bespoke media tooling for clients across the UK. Whether it's karaoke, captioning, or a completely custom rendering pipeline, [check my pricing](https://ajthe.dev/ai-solutions/#pricing) or [get in touch](https://discord.gg/d39aaZXAjh). For the grounded-documentary cousin of this same in-browser rendering approach, see my [DirectoryLM case study](https://ajthe.dev/case-studies/posts/directorylm-explainer-engine.html), or my [AJVV Studio TTS work](https://ajthe.dev/case-studies/posts/vibevoice-api-expansion.html) for the voice side of the media stack. --- **Written by AJTheDev** — North London developer. AI, FiveM, and web stuff. No bullshit. [Full story here](https://ajthe.dev/about/). Got an audio or video automation problem? Explore my [AI services](https://ajthe.dev/ai-solutions/), grab the [Karaoke video product on Gumroad](https://ajthedev.gumroad.com/l/karaokevids), join the [Discord](https://discord.gg/d39aaZXAjh), or use the [contact page](https://ajthe.dev/contact/).