--- title: "Expert Financial Data Annotation for AI" description: "CFA-level cases leave no room for approximation, and only one candidate in five passes. We built expert validation for meaning, calculations, and language." url: "https://unidata.pro/cases/expert-financial-data-annotation-for-ai/" date_modified: "2026-06-01T17:39:17+03:00" language: "en-US" --- **The Challenge** ----------------- The client needed annotation and validation of financial queries and model-generated responses. The material included complex financial cases with calculations, specialized terminology where domain understanding mattered as much as language level, and multi-step model solutions. Experts were required to: - assess the correctness of each query (both linguistically and economically) - validate every step of the model's response - identify errors in calculations and logic - evaluate terminology accuracy - deliver a final verdict on each answer. The CFA component added further constraints: tasks were in Russian, structured at examination level comparable to an international certification standard, and required narrower specialization than the FinQA track. A financial analysis pilot with even deeper domain requirements is currently in progress. **Key Challenges** ------------------ The candidate pool was extremely narrow — the role required both economics expertise and specialized vocabulary. Hiring conversion ran at roughly 20–25%. The operational team had no in-house domain knowledge, which made independent validation of expert decisions impossible and created heavy reliance on the client for interpreting task requirements. Designing the test assignment presented a separate problem: it could not be created without involving domain experts from the outset. **The Solution** ---------------- ### **Expert Recruitment** Candidates were drawn from economists, finance and analytics professionals, and specialists with verified English proficiency. The test assignment was developed with a domain expert and modeled real project cases. Selection criteria prioritized quality of reasoning and command of the subject area over throughput. This produced a core team of 8 experts for FinQA, which was later expanded to 14 for the CFA track. ### **Workflow Organization** It was clear from the start that the process needed a reliable mechanism for resolving edge cases, given that the operational team could not adjudicate expert decisions independently. The solution was a centralized document where experts logged ambiguous cases with examples. These were escalated to the client, and responses were distributed back to the full team. For time-sensitive issues, direct communication channels were used. ### **Annotation Process** Each task followed a fixed sequence: query review covering both language and economic meaning, step-by-step response validation, analysis of calculations and logic, terminology check, and final assessment. Quality control used a three-annotator overlap per task, with tag and score comparison to ensure inter-annotator consistency. ### **Scaling Expertise** On the CFA track, the initial pool was deliberately narrow given the certification-level subject matter. Senior experts trained the broader team, which made it possible to scale without compromising quality. The financial analysis pilot confirmed that deep within-domain specialization is a prerequisite, not an option, for projects of this type. | Stage | Input | Workflow Scope | Main Quality Checks | |---|---|---|---| | Project Setup | Client requirements, financial task formats | Task design, evaluation criteria, annotation guidelines | Task logic consistency, evaluation clarity | | Expert Onboarding | Candidate pool (finance background) | Recruitment, testing, interviews, onboarding | Expertise depth, language precision | | Annotation Execution | Financial Q&A tasks (FinQA, CFA-like) | Step-by-step validation of answers, calculations, reasoning | Calculation accuracy, logical consistency | | Multi-Review Process | Annotated tasks | Cross-review by 3 experts, disagreement resolution | Consensus alignment, error detection | | Validation & Analysis | Reviewed datasets | Error classification, pattern analysis, guideline refinement | Result consistency, systematic error control | | Reporting & Iteration | Validated financial datasets | Weekly reporting, feedback loops, quality improvement | Trend accuracy, continuous quality alignment | ## Main Title Expert Financial Data Annotation for AI ## Description CFA-level cases, multi-step calculations, and professional English, all at once. 20–25% hiring conversion, no in-house domain expertise on the ops side. How do you maintain expert consistency when the domain leaves no room for approximation? When a task demands not just language proficiency but genuine financial knowledge, standard annotation stops working. We built an expert validation process that covered meaning, calculations, and professional English simultaneously. ## Hero **Data:** 7000 financial cases **Project Duration:** 3 months ## Progress - Results - Quote ### Progress - Steps **List of Steps:** - **Number of days:** 1–2 weeks — **Step Description:** Pilot & Expert Calibration - **Number of days:** 2–3 weeks — **Step Description:** Expert Hiring & Validation - **Number of days:** ongoing — **Step Description:** Annotation & Multi-Review - **Number of days:** weekly, ongoing — **Step Description:** Quality Monitoring & Iteration ### Results **List of Results:** - A working expert annotation model for financial AI delivered - A process built and sustained without in-house domain expertise on the operations side - Specialist knowledge successfully scaled across the team - Stable inter-annotator consistency achieved in a multi-annotator setup - Quality positively assessed by the client ### Quote **Quote:** Financial reasoning in AI is not built on volume, but on the consistency of expert judgment. Models improve when every answer is challenged, validated, and aligned across multiple reviewers. **Author:** Vladislav Barsukov **Position:** Head of SLM&LLM Annotation [Full list of this site's AI-readable pages](https://unidata.pro/llms.txt)