Summary: A worked, reproducible extraction: 1,000 queries against a black-box classifier produced a replica agreeing with the victim on 80.8% of inputs — using a deliberately different architecture.
Sources: raw/Model-theft/Stealing AI Models Through the API_ A Practical Model Extraction Attack.md (Praetorian, 2026-01-23)
Last updated: 2026-07-28
The method
“An adversary with query access to a ML model can steal the model’s underlying functionality by systematically querying it and using the outputs to build a replica model.” Three steps:
- Send crafted inputs to the target’s API
- Record responses to build input-output pairs
- Train a replica on the collected data
The key insight
“Soft probability outputs contain far more information than hard labels. For example, when a model that classifies shoes returns ‘80% sneaker, 15% ankle boot, 5% sandal,’ it reveals learned relationships between classes, which are relationships the attacker can abuse to train a highly effective replica.”
Confidence scores are not a UX nicety. They are a training signal you are handing out for free.
Results
Real target (a production CNN for image analysis, zero knowledge of architecture or weights): 100 images uploaded, predictions recorded as “expert labels,” a replica trained on the pairs. The authors note the scores “aren’t cutting-edge” but are “worrying given the limited resources employed” — the replica “learned to identify the same patterns as the proprietary system using only 100 queries.”
Reproducible demo (Fashion-MNIST): after 1,000 queries and 20 epochs, the replica reached 80.1% accuracy and — the metric that matters — an 80.8% agreement rate with the victim. “The stolen model reproduces the victim’s behavior on 4 out of 5 inputs.”
Three findings that change the threat model
1. Architecture doesn’t need to be known. “Our replica model uses a deliberately different, simpler architecture to demonstrate that model extraction works even when the attacker doesn’t know the victim’s architecture. The stolen model learns to mimic behavior, not replicate structure.”
2. The replica inherits the victim’s failures. “Even when both models are wrong, they’re often wrong in the same way. The stolen model has learned the victim’s decision boundaries, including its failure modes.” Confusion matrices show both struggling on the same class pairs.
This is what makes extraction a stepping stone rather than an endpoint — a replica that shares your failure modes is a white-box oracle for finding inputs that break you.
3. Extraction success varies by class. “Simple, distinctive categories (Trouser, Bag, Sneaker) show >90% agreement. Ambiguous categories (Shirt, Pullover) show lower agreement, but still track the victim’s confusion patterns.”
The architectural flaw
“Many organizations assume that keeping model weights private is sufficient protection… But this creates a false sense of security. In reality, behavior is the model. Every query-response pair is a training example for a replica.”
This is the sentence to take away. It reframes Code of Practice Appendix 4 — 256-bit encryption, attested TEEs, copy registries — as necessary but addressing only one of two exfiltration paths. Appendix 4 protects the parameter file. It does nothing about the API handing out the model’s behaviour one query at a time.
Downstream uses
- IP theft — “Months of R&D, proprietary training data, and domain expertise can be stolen through API access.”
- Adversarial attack development — “Stolen models enable white-box attack development against black-box APIs. Once you have a local copy, you can craft adversarial examples at leisure, then deploy them against the production system.”
- Cost arbitrage — reselling cheaper inference while free-riding on the original R&D.
- Competitive intelligence — “even imperfect extraction reveals what features a competitor’s model has learned to prioritize.”
The second is the security-relevant one. Extraction converts a black-box target into a white-box one, which is precisely the precondition PITAX T09 gradient-based attacks needs — those “require either white-box model access (for gradients) or the ability to send thousands of queries.”
Extraction is how an attacker who has neither obtains the former.
Mitigations
- Rate limiting — “Our attack used 1,000 queries, a number that should trigger anomaly detection.”
- Output perturbation — calibrated noise on confidence scores, degrading distillation while preserving utility.
- Prediction truncation — “Return only top-k classes or hard labels instead of full probability distributions.”
- Behavioural monitoring — flag “rapid uploads of diverse images, sequential queries with slight variations, or systematic coverage of the input space.”
- Watermarking — “embed detectable patterns in model behavior that survive extraction,” providing evidence if a stolen model surfaces.
- Query pattern analysis — extraction traffic is “more systematic, more diverse, more evenly distributed across the input space” than legitimate use.
Output perturbation and prediction truncation both trade product quality for security, and the source is explicit that the trade exists. See model-theft-defenses.
The uncomfortable conclusion
“Model extraction attacks transform API access into model theft. The attack requires no special access, just the same query capability available to any legitimate user.”