Challenge
The project aimed to train models capable of automatically summarizing meetings and accurately distinguishing between different speakers. Our role was to prepare annotated audio data to power an AI bot designed to process and analyze conversations.
The client requested annotation of long audio fragments for model training, requiring precision and attention to detail at every step. Our tasks included:
- Segmenting long audio files with exact timestamps (e.g., from 00:01:23 to 00:01:45).
- Identifying speakers (e.g., Speaker-1, Speaker-2) and tagging unintelligible speech, breaths, and overlapping voices with dedicated labels.
- Transcribing the text following accurate segmentation.
Project challenges included:
- Long audio recordings (ranging from 16 to 60 minutes) with multiple speakers.
- The need to tag specific sounds separately.
- Strict accuracy requirements: no overlapping segments and precise time boundaries for each.
Solution
Preparation and workflow organization:
The project was split into two phases:
- Audio segmentation by speaker and sound type.
- Transcription of the segmented fragments.
We assembled dedicated teams for each phase: 5 annotators for segmentation and 5 for transcription, minimizing the risk of errors.
Training materials provided to annotators included:
- A detailed guideline document with tag examples.
- Video tutorials on handling complex cases and avoiding common mistakes.
- A Q&A table with clarifications from the client.
We also organized feedback sessions through video reviews of each annotator’s initial work.
Data annotation process:
- Annotators marked audio fragments with precise timestamps and assigned the appropriate tags.
- A separate team then transcribed the segmented audio, using special tags (e.g., [NAME] for names) to annotate entities within the text.
Quality control:
- We implemented a validation system with step-by-step checks for every file.
- Validators documented all issues in tracking tables with examples and explanations.
- In complex cases, a helpdesk was used for quick alignment with the client.
| Stage | Input | Workflow scope | Quality checks |
|---|---|---|---|
| Project setup | Client requirements, platform specs | Integration, task flow design, access configuration | System connectivityTask logic consistency |
| Participant onboarding | Annotator pool (5 segmentation + 5 transcription) | Recruitment, onboarding, instruction delivery, video reviews of initial work | Participant diversityInstruction clarity |
| Segmentation | Raw audio files | Speaker & sound-type segmentation with precise timestamps; tag assignment (e.g. [NAME]) | Boundary precisionNo overlap |
| Transcription | Segmented audio fragments | Text transcription with entity tags; separate team from segmentation to minimize error risk | Tag correctnessTranscript accuracy |
| Validation & QC | Annotated files | Step-by-step file validation; issue documentation in tracking tables; helpdesk for complex cases | Data completenessResult consistency |
| Reporting & iteration | Validated datasets | Weekly reporting, feedback loops, system improvement tracking | Trend accuracyContinuous alignment |
Pilot & setup
2 weeks
Participant onboarding
3 weeks
Annotation & iteration
Ongoing
Monitoring & reporting
Weekly, ongoing
The Results
- We delivered precise segmentation of 20 hours of complex audio.
Thanks to our two-step workflow and robust validation system, we achieved high-quality annotation that met the client’s requirements.
With recordings up to 60 minutes and four or five speakers talking over each other, the real risk wasn't missing a word, it was a one-second boundary error cascading through every downstream segment. Splitting segmentation and transcription into two independent teams was the decision that made everything else work. Validators weren't just catching mistakes: they were documenting patterns, which meant annotators stopped making the same errors by week two.
- Vladislav Barsukov
- Head of SLM&LLM Annotation
