Real-Time AI Broadcast Overlay & Captioning Engine

What the Prototype Proved

The working prototype demonstrated four things on one broadcast workstation:

• microphone-to-caption operation with local Whisper inference
• transcription without a cloud speech service
• operator-controlled captions, lower thirds, and speaker credentials
• transparent graphics composited into an OBS scene

My Role

As creator and developer, I designed and implemented the desktop interface architecture, Electron overlay surfaces, audio capture and transport integration, client transcript policy, local Whisper integration, and OBS broadcast workflow.

Three Architectural Decisions

1. Separate graphics from inference

The Electron/Vue application and FastAPI/Whisper service are separate processes. The graphics layer can evolve without owning the model runtime, and the transcription service can change without becoming responsible for broadcast presentation.

The trade-off is operational: a local multi-process system has more setup and coordination than a single application bundle.

2. Keep transcript acceptance policy in the client

The backend returns Whisper segments and metadata; the client decides which segments enter the visible caption queue.

That keeps presentation policy close to the UI that consumes it. The trade-off is that another client would need compatible policy or a shared policy layer.

3. Use Electron for the broadcast surface and OBS for composition

Transparent Electron windows provide the caption and lower-third surfaces while OBS remains responsible for the final scene.

That separates operator controls from what viewers see. It also makes the prototype desktop- and broadcast-workflow-specific rather than a browser-only product.

System Architecture

The system separates audio capture, WebSocket transport, local inference, transcript policy, overlay rendering, and final OBS composition.

The companion architecture research owns the source-level analysis of chunking, filtering, VAD status, audio-format calculations, and rendering behavior.

Broadcast Workflow

The operator-facing application controls captions and television-style context such as speaker credentials and lower thirds; transparent output surfaces are then composed with camera video in OBS.

Expand

Open Raw

Selected Technical Details

The microphone stream is converted to 16 kHz mono PCM and sent over WebSocket to the local FastAPI service. The backend accumulates fixed-duration chunks and runs Whisper inference off the async event loop. Returned segment metadata is evaluated in the client before accepted text enters the caption queue.

Those details are implementation evidence, not the center of this case study; the associated research entry documents the exact thresholds, derived bandwidth calculation, and renderer behavior.

Current Status

This is a working prototype, not a production distribution. The current source still contains prototype-quality choices I would revisit for sustained use: the legacy ScriptProcessorNode capture path, fixed-duration chunking, packaging and setup, automated testing, and the inactive PyAnnote VAD path.

The public repositories were published in August 2026, after the January–June 2024 initial development period shown here. The public history therefore demonstrates the current source, not the original development timeline.

The result is a complete local path from microphone input to on-air graphics, with inference, interface policy, rendering, and broadcast composition kept as distinct responsibilities.