Arabic Language Data Annotation for LLM Evaluation

The Task

A telecom client needed Arabic language data to validate internal AI tools.

Arabic is not a single operating language. Dialects vary so strongly that speakers from different regions may struggle to understand each other. At the same time, the client needed consistent, comparable results across tasks.

The scope included three parallel challenges:

• Verbatim transcription of Arabic audio with background noise, overlaps, laughter, and interruptions
• Evaluation of audio recordings after noise suppression, including safety assessment
• Linguistic evaluation of LLM generated Arabic texts based on a prompt and summary

Each task required native speakers. Some required dialect precision. All required strict linguistic judgment.

The Solution

Task Structuring

We separated this task into three independent pipelines:

• Speech transcription with explicit rules for non speech events
• Audio quality and safety evaluation with clear scoring logic
• LLM output evaluation with linguistic and semantic criteria

Each pipeline had its own guideline, examples, and quality signals. This avoided confusion and reduced subjective interpretation.

Dialect Mapping

Arabic is not a single working language, dialect differences are critical. That's why we worked with:

• Gulf dialects, including UAE and Saudi Arabia
• North African dialects, including Morocco and Algeria

We accounted for real linguistic behavior:

• English loanwords common in Gulf speech
• French insertions typical for North Africa
• Strong phonetic and lexical differences between regions

Annotators were matched to tasks strictly by dialect.

Annotator Sourcing

To control quality, we avoided mass recruitment. We quickly identified a common issue. Regional presence did not guarantee native language competence.

That’s why we:

• Sourced annotators manually via targeted LinkedIn search
• Validated native proficiency through test tasks, not profiles
• Required English for operational communication
• Matched annotators to tasks strictly by dialect

A recurring issue was false positives. People living in Arabic speaking countries but not native speakers. This was filtered out at the test stage. The final team was lean, predictable, and scalable.

Training and Calibration

Training was built around ambiguity, not theory.

• Test tasks revealed differences in how annotators interpreted transcription rules
• Feedback cycles aligned expectations quickly
• Special attention was given to LLM poetry evaluation, where grammar, logic, style, and prompt alignment all mattered

Annotators were trained to justify decisions, not just select labels.

In-Process Validation

Quality was monitored in real time.

• Ongoing reviews during production
• Immediate feedback on deviations
• Early detection of misunderstanding before it scaled

This minimized rework and protected timelines.

2 weeks

Setup & Preparation

1 week

Pilot Transcription & Evaluation

3 weeks

Core Annotation & Validation

1 week

Final Review & Delivery

The Results

• A reusable Arabic annotation framework across speech and LLM tasks
• Stable performance across multiple dialects
• Consistent quality despite linguistic complexity

You can’t treat Arabic as a single language. High-quality annotations require careful dialect selection, clear rules, and constant calibration.

Albina Romanova

Head of Speech Labeling & Data Generation