A liveness model can score high on its accuracy test and still get it wrong in production, flagging a real person as an attack because they're wearing glasses, or because the model has never seen a child's face before. Buying more training data before finding out where a model actually fails means paying to fix problems nobody has identified yet. How do you find them first?
The Problem
A Singapore-based identity verification provider already had a working face-biometric model and came to Unidata to source more training data for it. The obvious move is to buy every available dataset and feed it into the next training round. It is also the wasteful one: if the model already handles half of these scenarios well, that part of the spend buys nothing.
So the brief was different. Before any purchase, run an AI Model Check: test the client's model against a large, structured video set, find out exactly where and why it fails, and let those results decide what data is actually worth buying.
Solution
Test Setup
The check ran the client's model against three datasets, 135,978 videos combined:
- iBeta 1 and iBeta 2, the client's main liveness test sets, built around adult faces
- iBeta Kids, a separate set built around children's faces, up to age 17
The model's output was not reduced to a single accuracy score. It was broken down by attack type, face attribute, and age group, so weak spots would show up on their own instead of getting averaged out.
The Attribute Blind Spot
Broken out by category, the error rate was uneven: up to 5% on standard presentation attacks, up to 15% on silicone masks, and up to 25% on real faces from rare or underrepresented ethnicities. One failure mode stood out in particular: a real face wearing an attribute (glasses, jewelry, and similar items) was misclassified as a biometric attack in about 20% of these cases. The check traced this to how the model was trained rather than to a random error:
- during training, it saw far more attack examples than real faces
- as a result, it learned to expect a real face to look close to ideal
- glare, an accessory, or an unusual angle is enough of a deviation to get read as an attack
This is not unique to the client's model. It is a pattern the check turns up on biometric liveness models generally.
Children's Biometrics
The same test was run separately against the iBeta Kids dataset. A model trained mostly on adult faces treats a child's face as an anomaly for the same reason it treats glasses as one: it simply does not match what the model learned a real face looks like.
Unidata includes this age-group test as part of the Model Check whenever a children's dataset is available.
From Findings to Data Purchase
The 20% figure gave the client a specific, named gap instead of a vague sense that the model "could be better." Based on it, the client bought additional data of real faces wearing attributes (glasses, jewelry, and similar items), closing that exact gap rather than repurchasing all three original datasets.
Process
| Phase | Input | Scope of Work | Quality Control |
| Model & Requirements Intake | Client's liveness model, target attack scenarios | Access setup, defining test scope across face, attribute, and age datasets | Test scope matches client's model and use case |
| Test Dataset Assembly | iBeta 1, iBeta 2, iBeta Kids (135,978 videos) | Running the client's model against all three datasets, structuring output by attack type, attribute, and age group | Coverage confirmed across attack categories and age groups |
| Error Analysis | Model inference results | Accuracy breakdown by category; isolating real-face-with-attribute misclassifications | Systematic false-positive patterns identified, not just an aggregate score |
| Weak Spot Report | Validated error findings | Documenting strong and weak spots, attack-type distribution, data gaps | Findings are specific and reproducible |
| Targeted Data Sourcing | Weak spot report | Recommending and preparing datasets that address the identified gaps | New data maps directly to a documented weakness |
| Delivery & Handoff | Final report, sourced dataset | Packaging the report and data with a Docker container for reproducibility, walking the client through the findings | Client sign-off on the findings and the resulting data purchase |
The Results
- An error map by attack type and subject group, not one blended accuracy score
- A ~20% misread rate isolated to one subgroup: real faces wearing attributes
- An adult-face bias caught before it reached production, via a dedicated children’s-face test
- A data purchase aimed at those three gaps, not a repeat buy of everything
We see this blind spot on almost every liveness model we check. It's trained mostly on attacks, so it barely learns what a normal face looks like. Someone puts on glasses, or the room lighting is a little off, and the model calls them fake.
- Elizabeth Karnaukhova
- Datamarket Project Manager
