Canary

Adversarial prompt-injection detection across LLM tool-use pipelines

Research Lead Marek Seablom-MichelTerm 2026–2027Research Team Chace Lai, Dhruv Jadhav, Miles Schulist, Samson Chang, Johan Jijo

Prompt injection involves an attacker who hides instructions in content an AI receives, which can make the agent do malicious activity; it’s rated as the #1 risk on OWASP for AI applications, and work presented at NeurIPS finds that even top-tier models — Claude, ChatGPT, and Gemini among them — are deceived by simple, low-effort human-written injections, which partially succeed in up to 86% of cases. With Gartner projecting a 700% rise in enterprise applications carrying task-specific AI agents, if prompt injection can’t be properly addressed, corporations can be exploited, critical infrastructure can be penetrated, and technological development will slow down.

Addressing prompt injection can be divided into two paths: gaining awareness of possible attacks and creating software to help systems defend against attacks that have already been made. CANARY sets out to create measurement tools to help companies test their systems against prompt injection, bring light to unique attacks, and defend against them.

The methods developed in this research can:

  • Protect the next generation of AI-assisted software development in large industries and startups by creating new protection against prompt injection attacks.
  • Provide vital infrastructure to industries by advancing measurement strategies to more accurately benchmark models.
  • Prepare the growing amount of agents embedded into people’s emails, phonebooks, and lives by discovering new prompt injection attacks that could take place.

Literature Review

Works Cited

  • Evtimov, Ivan, et al. “WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks.” Advances in Neural Information Processing Systems 39, Datasets and Benchmarks Track, 2025. PDF.
  • Gartner, Inc. “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025.” Gartner Newsroom, 26 Aug. 2025. gartner.com.
  • Khan, Adil, Khaled AlKhanbashi, and Azza Mohamed. “Evaluating Indirect Prompt Injection Defenses in Tool-Using LLM Agents: Security, Utility, and Replication.” Computers, vol. 15, no. 9, 2026, article 570. doi.org/10.3390/computers15090570.
  • OWASP GenAI Security Project. “LLM01:2025 Prompt Injection.” OWASP Top 10 for Large Language Model Applications, 2025. genai.owasp.org.
  • Wang, Peiran, et al. “The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis.” arXiv, 2026. arxiv.org/abs/2602.10453.