Speaker Captions

A local captioning app that combines speech recognition and face tracking to label who is speaking in a video.

Speech recognition and computer vision

Captions with
speaker labels.

In this project, I worked on combining speech recognition with face tracking to generate captions and associate them with the person speaking. The app runs locally and supports both live camera input and recorded videos. In my idea, this app was intended to assist imparied people to understand who is talking and keep track of the speaker visually.

Watch the demo Code on GitHub
  • Python
  • Computer vision
  • Speech recognition
  • Electron
Speaker Captions / offline output24 SEC
Example output This clip shows 24 seconds of a Giuseppe Conte speech processed by the app. The highlighted person and their captions share a colour. The known voice stays linked to Person 1’s tracked face, keeping the rectangle and caption consistent through brief visual-score fluctuations. The audio and captions are in Italian.

Features

Live captions and video processing

LIVE

Camera and microphone

The Mac app highlights the person speaking and displays their captions in a transcript. It uses face and voice features to keep speaker labels consistent during a session, including when someone leaves and returns to the frame.

Vosk · MediaPipe · SFace · ECAPA

OFFLINE

Recorded videos

For recorded videos, the app exports an MP4 with highlights and captions, a subtitle file, and JSON files with the intermediate results. A review page helps check the transcript alongside the video.

Whisper · ECAPA · MediaPipe · Light-ASD

Implementation

How it works

  1. 01

    Speech recognition

    Speech recognition produces the transcript and word timestamps. Voice embeddings are used to group speech from the same person, without identifying who they are.

  2. 02

    Face tracking

    Face tracking locates people in the video. For recorded videos, Light-ASD estimates which face is speaking. Live mode uses mouth movement together with evidence collected over recent frames.

  3. 03

    Speaker assignment

    The audio and video results are aligned by timestamp to assign each caption to a person. The app also handles cases where the speaker is off camera or the assignment is uncertain.

Development notes

Challenges and limitations

Much of the work went into speaker assignment. A listener moving their mouth can be mistaken for the speaker, small faces are difficult to track, and a person may keep talking after leaving the frame. I used replays of the example video to check the results and tune the live pipeline.

The transcript comes from audio, so the app does not read lips from silent video. Speaker assignment is still experimental and has mainly been evaluated on one recording. Overlapping speech, background noise, and small faces can lead to mistakes.

The app is still unreliable for multiple speakers, which is next step of development. It works, but it has difficulties in distinguishing speakers when they talk together or move out of the camera.

Local processing. Models are downloaded on first use and run on the machine. Speaker labels are kept within each recording or live session.

AI assistance. The code was created with Claude, with my supervision of the design.

Documentation and evaluation notes