Task
The client needed real-world spoofing attempts to test and improve their face recognition and liveness detection system. Participants were invited to actively try to bypass the system, rather than simply submit passive data.
The main attack types included:
- Print attacks using printed photos manipulated in different ways
- Replay attacks attempting access via replayed visual material on devices
Each attack attempt was evaluated by the client’s system and assigned a score, reflecting how convincing and successful the attempt was.
Key challenges included:
- Attracting and retaining participants willing to repeatedly test and “break” the system
- Ensuring high variability across attacks (devices, print quality, lighting, backgrounds)
- Coordinating technical integration between crowd platforms and the client’s interface
- Capturing detailed behavioral data without direct access to the client’s internal system
Solution
Preparation and technical setup
- Integrated with the client’s platform through an external execution interface, redirecting participants from the crowd platform to the client’s system
- Worked via an existing enterprise account to manage task launches and participant access
- Designed task flows that encouraged experimentation rather than one-off submissions
Attack execution process
- For print attacks, participants received a facial image, printed it, and physically manipulated it (bending, rotating, folding) while attempting to pass system checks
- Successful or near-successful attempts unlocked the option to return and retry, motivating participants to improve their techniques
- Replay attacks were collected in parallel and reached target volume faster, while print attacks required longer-term iteration
- Ensured broad diversity across:
- age groups
- ethnic backgrounds
- devices and cameras
- print materials and presentation styles
Monitoring and reporting
- Tracked participant behavior step by step: number of attempts, repeat participation, and attack outcomes
- Delivered weekly reports to the client, including:
- number of new attacks
- print vs. replay breakdown
- new vs. returning participants
- system response trends
- Held regular review calls to align on findings and system performance changes
| Stage | Input | Workflow Scope | Main Quality Checks |
|---|---|---|---|
| Project Setup | Client platform & anti-spoofing requirements | Integration via external interface, task flow design, access configuration | System connectivity / Task logic robustness |
| Participant Onboarding | Global contributor pool | Recruitment, onboarding, adversarial task briefing | Participant diversity / Instruction clarity / Attack readiness |
| Attack Execution | User devices, printed images, replay materials | Print & replay attacks, iterative submissions, retry-based engagement | Attack variability / Scenario realism / Adversarial complexity |
| Behavior Tracking | Attack attempt data | Tracking retries, repeat participation, attack progression | Data completeness / Iteration patterns / Strategy consistency |
| Validation & Analysis | System-scored attack data | Score review, performance analysis, false acceptance signal detection | Score reliability / Attack success signals / Edge-case identification |
| Reporting & Iteration | Validated attack datasets | Weekly reporting, trend analysis, continuous feedback loop | Trend accuracy / Model response shifts / Continuous performance alignment |
Pilot & Setup
2 weeks
Participant Onboarding
3 weeks
Attack Collection & Iteration
ongoing
Monitoring & Reporting
weekly, ongoing
The Results
- Generated a large and diverse set of realistic spoofing attempts over a two-year period
- Helped our client continuously refine and harden their biometric system
- Observed measurable improvement in system resistance as attack strategies evolved
- Built a sustainable feedback loop between real users and biometric security engineers
A system tested only in controlled conditions hasn't really been tested, the gaps always show up in production.
- Hanna Parkhots
- Data Collection Project Manager
