The Task
A retail client approached us with a clear goal:
Automate the process of monitoring grocery store shelves using neural networks.
They needed a dataset that would enable a machine learning model to detect and classify products on shelves in real time. The end use case?
- Measure the success of promotions
- Optimize shelf space
- Respond faster to stockouts
But there was a significant challenge:
Shelves were filled with a huge variety of products—different brands, categories, package designs, and frequent seasonal updates. Traditional annotation workflows weren’t going to cut it.
The Solution
Structuring the Work
We split our team into two focused groups:
- Product Research Team:
This team created a taxonomy of product categories. They studied the client’s inventory, researched visual differences between product types, and developed detailed classification criteria for annotators. - Annotation Team:
Using these guidelines, annotators worked on labeling every image with high precision, tagging product types, positions on the shelf, and packaging variations.
Tooling and Workflow Setup
- We used a combination of internal QA dashboards and custom labeling tools to track accuracy.
- A feedback loop was built in—researchers could refine guidelines based on edge cases found by annotators.
- Weekly calibration sessions ensured that annotators and researchers were always aligned.
Quality Assurance
- A dual-pass review process was implemented: all images were reviewed by a second annotator.
- Random samples were escalated to experts for manual audit.
- Discrepancies were analyzed to refine both training and guidelines.
| Stage | Input | Workflow Scope | Main Quality Checks |
|---|---|---|---|
| Guidelines & Setup | Platform policies, sample queries | Define intents, annotation rules, verification logic | Guideline clarity / Coverage of key intents |
| Pilot Annotation | Sample queries | Test verification logic, refine workflow, early feedback | Annotation accuracy / Logic validation |
| Full Annotation | User messages across categories | Annotate intents, differentiate response types, match listing content | Consistency / Context-aware labeling |
| Validation | Annotated dataset | Quality review, anomaly detection, validator collaboration | Accuracy / Alignment with project rules |
| Final Delivery | Validated intent dataset | Consolidation, final QA, submission to client | Dataset completeness / Intent coverage |
Guidelines & Workflow Setup
5 days
Pilot Annotation & Verification Logic Testing
10 days
Full Annotation Cycle
2 weeks
Validation, QA & Final Delivery
2 weeks
The Results
- 40% Cost Reduction: By streamlining roles and using task specialization, we lowered total project costs significantly.
- High-Precision Dataset: The annotated images provided clean, structured training data for the client’s neural network, supporting accurate real-time shelf analytics.
- Better Business Insights: The client could now evaluate promotional campaign results in real time, detect planogram violations, and improve in-store execution.
Accurate intent annotation turns fragmented user messages into structured insights, enabling AI to respond precisely, contextually, and at scale.
- Vladislav Barsukov
- Head of SLM&LLM Annotation
