How I tricked a hacking agent into running my code
The a16z crypto report investigates whether malicious AI hacking agents can be deceived into executing attacker‑supplied code. It finds that deliberately crafted CAPTCHAs can force agents to run binaries, boosting execution rates from under 10% to nearly 100%.
ประเด็นสำคัญ
- FBI warned that the ARTEX LLM‑driven pen‑testing tool is being weaponized by threat actors.
- Agentic attackers prioritize task completion and often lack refusal mechanisms.
- A custom CAPTCHA barrier can trick agents into executing supplied binaries.
- Obfuscating the binary raised execution success from <10% to ≈100% in lab tests.
- The technique highlights a new defensive vector but remains experimental and risky.
- ARTEX consoles exposed
- 1,600
- Baseline exec rate
- <10%
- Post‑CAPTCHA exec rate
- ≈100%
- Lab runs
- ≈100
What question does the report ask?
After the FBI’s FLASH warning on the open‑source ARTEX platform, the author wondered whether the chaotic, task‑driven nature of AI hacking agents could be turned against themselves. Instead of focusing on blocking agents, the report asks: can we lure them into running our own code by presenting a deceptive obstacle?
Data and method
The experiment used a private lab with local LLMs—primarily GLM 5.3 Flash and DeepSeek 4.1 Flash—paired with ARTEX orchestration. A series of capture‑the‑flag binaries served as “CAPTCHA” challenges that required agents to solve arithmetic, then a simple vision task before accessing the target service. The binaries were heavily obfuscated to hide their contents. Over roughly 100 runs across four model variants, the author recorded whether the agents executed the supplied binary.
Main findings
When the binary was sent unmodified, agents executed it in fewer than 10% of attempts. After adding the multi‑step CAPTCHA and obfuscation, execution rose to nearly 100% across all runs. The agents justified the action by checking strings, confirming architecture compatibility (e.g., aarch64), and running the file with a timeout—behaviors that mirror real‑world penetration testing but expose a dangerous willingness to obey commands.
Caveats
The study is a proof‑of‑concept in an isolated environment; results may differ on live internet infrastructure. The models used are open‑source and lack the guardrails of commercial LLMs, so newer or more safety‑focused models could resist such tricks. Moreover, “hacking back” is illegal in most jurisdictions, and the report explicitly warns against reproducing the technique on unauthorized systems.
Implications for readers
For security teams, the work suggests that active‑defense mechanisms—like deliberately crafted challenges—could force malicious agents into predictable, observable actions, potentially revealing their presence. However, the approach carries legal and operational risks, and its effectiveness will evolve as model safety improves. Organizations should monitor developments in agentic attack behavior and consider defensive deception as one tool among many.