Task
The client requested the annotation of legal documents using Label Studio. The goal was to highlight key legal entities (such as seller, buyer, alienated right, and representative) and accurately establish relationships between them.
Key challenges included:
- Complex legal language and conditional constructions
- High sensitivity to errors — missing or misinterpreting even one word could distort the annotation
- Lack of standardization across documents (contracts, inheritance certificates, etc.)
Solution
Preparation and workflow organization:
- Developed detailed technical guidelines tailored to 20+ annotation scenarios
- Created a document with Q&A updates from the client to clarify edge cases
- Provided annotated examples with screenshots for each entity type
- Recorded training videos and provided personalized video feedback on initial tasks
- Implemented a helpdesk-style internal communication channel for quick resolution of questions
Data annotation:
- Annotators identified core legal entities and manually mapped relationships (e.g., linking an alienated right to both the seller and the asset)
- Ensured structural consistency (e.g., linking a representative to the main party in the transaction)
- Used Label Studio for precise control over annotation fields and linkage logic
Quality control:
- Each document was reviewed by validators, who provided structured feedback
- Errors were described in detail in validation spreadsheets and returned for correction
- Continuous feedback loops improved overall annotation quality over time
Pilot & Team Selection
7 days
Guidelines & Client Alignment
10 days
Collection & Annotation
28 days
QA & Final Delivery
7 days
The Results
- Annotated over 6,000 legal documents with high precision, ensuring correct entities and relationships
- Improved annotator skills through continuous training, example reviews, and weekly calibration sessions
- Established a reliable workflow for complex legal logic, allowing consistent annotation across diverse document types
- Maintained quality and consistency across a high-complexity dataset
Legal documents are highly sensitive to even minor errors. Carefully selecting experienced annotators and maintaining continuous feedback with the client was crucial to achieving precise entity and relationship mapping across 6,000 documents.
- Albina Romanova
- Head of SLM&LLM Annotation
