Architecture Notes: Audio Chunking, Transcription Filtering, and Real-Time Speech UI Systems

The speechBubbles project case study establishes the working prototype, creator role, broadcast workflow, and system-level responsibility boundary. This paper begins inside that boundary and inspects four implementation paths: audio-format reduction, chunking, transcription filtering, and async inference/render scheduling.

Real-time speech UI is a pipeline problem. Capture, format reduction, buffering, inference, filtering, state, and rendering introduce different constraints, so the architecture is easier to reason about when those responsibilities remain separate.

The Audio Sample Rate Dilemma

Consumer audio hardware and browser AudioContext instances commonly operate at 44,100 Hz or 48,000 Hz using floating-point sample buffers.

Sending 48 kHz stereo 32-bit float audio directly would produce about 384 KB/sec of raw sample data before protocol overhead:

48,000 samples/sec × 2 channels × 4 bytes/sample = 384,000 bytes/sec.

The Client-Side Downsampling Solution

In CaptionControl.vue, downsampling is performed before packet transmission:

function downsampleBuffer(buffer, inputSampleRate, outputSampleRate = 16000) {
  if (inputSampleRate === outputSampleRate) return buffer;
  
  const sampleRateRatio = inputSampleRate / outputSampleRate;
  const newLength = Math.round(buffer.length / sampleRateRatio);
  const result = new Float32Array(newLength);
  
  let offsetResult = 0;
  let offsetBuffer = 0;
  
  while (offsetResult < result.length) {
    const nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio);
    let accum = 0, count = 0;
    
    for (let i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) {
      accum += buffer[i];
      count++;
    }
    result[offsetResult] = count > 0 ? accum / count : 0;
    offsetResult++;
    offsetBuffer = nextOffsetBuffer;
  }
  return result;
}

The filtered float array is then clamped and converted to signed 16-bit integers (Int16Array). At 16 kHz × 16-bit mono, that format is approximately 32 kB/s (32,000 bytes/s) before WebSocket overhead—a mathematically derived reduction of about 91.7% compared with the 48 kHz stereo 32-bit-float example above.

Buffered Chunking: Delay vs. Linguistic Context

Speech transcription needs enough surrounding audio to resolve word boundaries and linguistic context. That creates a practical trade-off:

The implementation of that default is visible in fastapi_app.py:

chunk_length_bytes = int(client.config.get("chunk_length_seconds", 3)) * client.sampling_rate * client.samples_width

if len(client.buffer) >= chunk_length_bytes:
    client.scratch_buffer.extend(client.buffer)
    client.buffer.clear()
    asyncio.create_task(self.process_buffer(client, websocket))

After transcription finishes, scratch_buffer is cleared. Because processed samples are not intentionally retained and reused across overlapping windows, describing this as a sliding window would overstate what the current source implements.

Handling Low-Confidence Transcription Output

Continuous microphone input can cause Whisper to emit undesirable text during silence, ambient noise, or otherwise low-confidence audio. The public source currently addresses this primarily on the client after transcription.

Current Backend Behavior

whisper-api includes a VADFactory, PyAnnote dependencies, and a lazy get_vad() helper. However, the current public process_buffer() implementation does not invoke that VAD pipeline before model.transcribe().

Client-Side Probabilistic Filtering

In CaptionControl.vue, returned segments are evaluated against Whisper metadata:

function filterSegments(segments) {
  for (let x = 0; x < segments.length; x += 1) {
    const seg = segments[x];
    const noSpeech = seg.no_speech_prob ?? 0;
    const temp = seg.temperature ?? 0;
    if (noSpeech <= 0.6 && temp <= 0.8 && seg.text?.trim()) {
      store.captions.push(seg.text.trim());
    }
  }
}

Segments above those thresholds are discarded by the client. This helps suppress likely hallucinated or low-confidence text before it reaches the caption queue, but it does not guarantee that all phantom output is eliminated.

Async Inference & UI Rendering

The backend uses asyncio.to_thread() around model.transcribe(), moving blocking inference off the async event loop. The same principle—keep blocking work off the responsive path—applies in the renderer.

CaptionsView.vue measures caption geometry after Vue’s nextTick(), then uses nested requestAnimationFrame() callbacks to apply the transition styles:

const performAnimation = (multiplier = 1) => {
  requestAnimationFrame(() => {
    requestAnimationFrame(() => {
      reference.$el.style.transition = `all ease-in-out ${animationTiming * multiplier}ms`;
      reference.$el.style.opacity = x >= messageRefs.value.length - 1 ? 1 : 0;
      reference.$el.style.transform = `revert`;
      setTimeout(() => {
        if (reference.$el) reference.$el.style.transition = 'unset';
      }, animationTiming * multiplier);
    });
  });
};

The current prototype still performs synchronous geometry reads when captions mount, including a second getBoundingClientRect() after style changes. That is acceptable as implementation evidence for the current prototype, but it is not presented here as an optimized rendering strategy; a production pass should batch geometry reads before DOM writes to reduce forced-layout risk.

Architectural Takeaways & Future Directions

The most defensible principle is specific: capture, blocking inference, acceptance policy, and rendering are easier to inspect when their ownership remains explicit. That separation does not establish a latency or reliability improvement by itself; it makes the different failure modes and optimization targets easier to locate.

The next experiments follow those boundaries: wire the existing VAD path into the active transcription flow, compare chunk-duration strategies rather than treating the current default as optimal, and batch caption geometry reads before DOM writes.

For the practical implementation and video demonstration of this architecture in action, visit the speechBubbles Project Case Study.