The Task
A telecom client needed Arabic language data to validate internal AI tools.
Arabic is not a single operating language. Dialects vary so strongly that speakers from different regions may struggle to understand each other. At the same time, the client needed consistent, comparable results across tasks.
The scope included three parallel challenges:
- Verbatim transcription of Arabic audio with background noise, overlaps, laughter, and interruptions
- Evaluation of audio recordings after noise suppression, including safety assessment
- Linguistic evaluation of LLM generated Arabic texts based on a prompt and summary
Each task required native speakers. Some required dialect precision. All required strict linguistic judgment.
The Solution
-
- 01
-
Task Structuring
We separated this task into three independent pipelines:
- Speech transcription with explicit rules for non speech events
- Audio quality and safety evaluation with clear scoring logic
- LLM output evaluation with linguistic and semantic criteria
Each pipeline had its own guideline, examples, and quality signals. This avoided confusion and reduced subjective interpretation.
-
- 02
-
Dialect Mapping
Arabic is not a single working language, dialect differences are critical.
That’s why we worked with:
- Gulf dialects, including UAE and Saudi Arabia
- North African dialects, including Morocco and Algeria
We accounted for real linguistic behavior:
- English loanwords common in Gulf speech
- French insertions typical for North Africa
- Strong phonetic and lexical differences between regions
Annotators were matched to tasks strictly by dialect.
-
- 03
-
Annotator Sourcing
To control quality, we avoided mass recruitment. We quickly identified a common issue. Regional presence did not guarantee native language competence.
That’s why we:
- Sourced annotators manually via targeted LinkedIn search
- Validated native proficiency through test tasks, not profiles
- Required English for operational communication
- Matched annotators to tasks strictly by dialect
A recurring issue was false positives. People living in Arabic speaking countries but not native speakers. This was filtered out at the test stage. The final team was lean, predictable, and scalable.
-
- 04
-
Training and Calibration
Training was built around ambiguity, not theory.
- Test tasks revealed differences in how annotators interpreted transcription rules
- Feedback cycles aligned expectations quickly
- Special attention was given to LLM poetry evaluation, where grammar, logic, style, and prompt alignment all mattered
Annotators were trained to justify decisions, not just select labels.
-
- 05
-
In-Process Validation
Quality was monitored in real time.
- Ongoing reviews during production
- Immediate feedback on deviations
- Early detection of misunderstanding before it scaled
This minimized rework and protected timelines.
The Result
A reusable Arabic annotation framework across speech and LLM tasks
Stable performance across multiple dialects
Consistent quality despite linguistic complexity
