Training AIs to Solve CAPTCHAs

Summary

Researchers built a CAPTCHA solver that pairs a fine-tuned YOLOv8 detector with an open-weight vision-language model behind a confidence-based router, operating only from screenshots and OS input events. It reaches 85.4% overall and 84.2% macro accuracy across 16 classes, and the detector learns new categories from the model's answers without human annotation. Adversarial perturbations that drive the detector to 0% accuracy are recovered through the same loop, which the authors argue means visual CAPTCHA defenses understate how quickly an adaptive solver returns.

Related items

• InfoScrewtape: Adversarial Transcript Editing for Monitor Red-TeamingSimilar attack · Redwood Research Blog
• InfoEnhancing Visual Task-Redirection Attacks on Vision-Language-Action Models via Closed-Loop OptimizationSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoSensitive area-guided adversarial attack via Transformer-based generative networkSimilar attack · OpenAlex (peer-reviewed AI security)
• LowA Valid Watermark Is Not a Valid Identity: Spatial-Spectral Adversarial Face Evasion with Watermark PreservationSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoTopology-aware dynamic co-evolution for robust graph learningSimilar attack · OpenAlex (peer-reviewed AI security)