01 // DATA SOURCESWhat every row stands on
Four kinds of source
Each one linked from the row it supports
ATTACK EVIDENCEPrimary reports
32 reports from 22 publishers: security vendors, IR firms, CERTs, government agencies and AI labs that saw the activity themselves. A harness is tied to an attack only when one of these names it. News stories and forum posts are leads, never evidence.
REPO FACTSGitHub API
Stars, license, last push and archived status for 153 repos, pulled from the GitHub REST API on every refresh. A repo that disappears fails the refresh instead of quietly dropping off the page.
USE TAGSProject READMEs
Outlined use tags quote the project's own README sentence by sentence, so they are the project's claim, not a tested result. Every refresh re-reads each README and fails if a quoted sentence is gone. 331 README sentences back the tags today.
DISCOVERYLists and papers
Candidates come from GitHub search, then from curated awesome lists and academic papers read for repos the search missed. Discovery only puts a repo on the list to check; the roster rule decides whether it stays. These are the lists and papers that gave the roster a candidate; section 03 has every reference.
02 // COLLECTIONHow a row gets made
Refresh fails past 21 days
without a new evidence review
- Discover
Search GitHub by many phrasings and topics, in English and Chinese, then mine curated lists and papers for repos the search missed.
- Apply the roster rule
Public on GitHub, self-described as an offensive security agent, harness or skill pack with an LLM in the loop, an OSI license (or a noted override), and 100 stars or more (or a primary report that names it). Source-available and commercial projects are left out.
- Pull the facts
fetchField.tspulls stars, license, last push and archived status for every repo and writes them to the dataset with the date. - Tag from the README
A reader finds the README line that states each use. Code copies the line verbatim, so no tag rests on a paraphrase.
- Research attacks
Read each primary report itself, not a summary of it. Record the actor, what the harness did, whether a breach is stated, and the report's own sentence for each attack tag.
- Assign the tier
Confirmed: shown in a real operation. Observed: aimed at real targets or found on attacker infrastructure, no breach stated. Experimentation alone stays off the attack table.
- Independent review
A second model that did not do the research re-reads the report behind each new or changed attack row and checks every quote against it. It also checks the README quotes on new harnesses: all exploitation, exploit-development and post-exploitation tags, and a sample of the rest. Each finding is adopted or rebutted in the project record.
- Build behind gates
The page, data.json and status.json render from one dataset. The build refuses stale evidence, dead internal links and rows missing a source, a description or a README sentence, and the deploy runs the tests first.
03 // REFERENCESLists, benchmarks, papers and reports
72 sources, each one opened when it was added
Marked discovery: mined for roster candidates
Curated lists 20 · 10 marked discovery
| Awesome-AI-Hacking-Agents (EvanThomasLuke) ↗ discovery | A list of AI hacking agents with links to each project. |
|---|---|
| awesome-ai-offensive-security (yeyintminthuhtut) ↗ discovery | AI agents and tools built for offensive security, plus tools for attacking AI systems. |
| awesome-ai-pentest (insidetrust) ↗ discovery | Open-source AI-assisted pentesting tools, frameworks, CTF agents and benchmarks. |
| awesome-ai-pentesting (skyvanguard) ↗ discovery | AI-powered pentesting tools, frameworks and MCP servers for autonomous security testing. |
| awesome-ai-security (gmh5225) ↗ discovery | AI security materials for pentesters, bug hunters and security researchers. |
| awesome-ai-security-tools (scadastrangelove) ↗ discovery | Public-source, research and commercial tools for AI security and AI-assisted security work, from pentest agents to AI code scanners. |
| Awesome-Hacking-with-AI (frangelbarrera) ↗ discovery | A guide to AI for security and security for AI, covering agent safety, MCP security and AI red-teaming. |
| Awesome-LLM4Security (liu673) ↗ discovery | Resources and tools for using LLMs across cybersecurity. |
| Awesome-Offensive-AI-Agentic-Landscape (Yeti-791) ↗ discovery | Open-source projects, papers, benchmarks and commercial products for AI pentesting and autonomous offensive agents. |
| awesome-security-agent-harnesses (Ed-Marcavage) ↗ discovery | AI pentesting tools and security agent harnesses for code audit, vulnerability discovery, fuzzing and reverse engineering. |
| Awesome-AI-for-Hacking-Cybersecurity (Laprovittera) ↗ | A large Spanish-language list of AI agents, tools, frameworks and papers for hacking and security, merged from eight community lists. |
| awesome-autonomous-hacking-agents (vitorfhc) ↗ | Papers on building autonomous security bots: offensive AI, multi-agent design, memory and tool use. |
| awesome-cyber-ai-arsenal (pazo01) ↗ | Offensive, defensive and AI/LLM security tools in one curated collection. |
| awesome-cyber-security-mcp (MorDavid) ↗ | An index of MCP servers that connect AI models to security tools. |
| awesome-llm-cybersecurity-tools (Tenable) ↗ | LLM tools for cybersecurity research. |
| Awesome-LLM4Cybersecurity (tmylla) ↗ | An overview of how LLMs are used in cybersecurity. |
| Awesome-LLMs-for-Vulnerability-Detection (huhusmang) ↗ | An index of research on LLMs for finding software vulnerabilities, with datasets, benchmarks and surveys. |
| awesome-mcp-security (Puliczek) ↗ | Resources on the security of Model Context Protocol (MCP) servers and clients. |
| awesome-offensive-security-ai (ox01024) ↗ | Offensive security AI work: papers, organizations, libraries and benchmarks. |
| offensive-ai-compilation (jiep) ↗ | Resources that cover offensive uses of AI. |
Benchmarks 18
| AIRTBench ↗ | Code for AIRTBench, which measures how well language models do autonomous AI red teaming. |
|---|---|
| AutoPenBench ↗ | A penetration testing benchmark for generative agents, with vulnerable containers you can add to. |
| BountyBench ↗ | A bug bounty benchmark that tests AI agents on real-world vulnerabilities. |
| CAIBench ↗ | A meta-benchmark for evaluating cybersecurity AI agents. |
| CTFusion ↗ | A streaming framework that benchmarks LLM agents on live CTF competitions. |
| CVE-Bench ↗ | Real web application vulnerabilities for testing whether AI agents can exploit them. |
| Cybench ↗ | 40 CTF tasks from four competitions, with subtasks, for scoring how well agents handle cybersecurity challenges. |
| CyberGym ↗ | Berkeley RDI's observatory of benchmarks on how well AI agents find, reproduce and exploit real vulnerabilities. |
| EthiBench ↗ | Ethiack's AI pentesting benchmark framework, the code behind the From Controlled to the Wild paper. |
| evmbench ↗ | A benchmark and harness for finding and exploiting smart contract bugs, made with OpenAI. |
| ExploitBench ↗ | Measures how far an agent climbs, from reaching vulnerable code to triggering the bug to arbitrary code execution. |
| ExploitGym ↗ | Real-world vulnerabilities for testing whether AI agents can develop working exploits. |
| inspect_cyber ↗ | An extension to the Inspect framework for running agentic cyber evaluations. |
| NYU CTF Bench ↗ | 200 dockerized CSAW CTF challenges across six categories for testing LLM agents, with a leaderboard. |
| PentestEval ↗ | Scores LLMs on each stage of a pentest, from gathering weaknesses to writing exploits for live containers; a small public demo release. |
| PurpleLlama CyberSecEval ↗ | Meta's cybersecurity benchmarks for LLMs, part of the PurpleLlama toolset for assessing LLM security. |
| SEC-bench ↗ | An automated benchmark of LLM agents on real-world software security tasks (NeurIPS 2025). |
| XBOW validation-benchmarks ↗ | Web security challenges used to score autonomous pentest agents; its authors now call them saturated and keep them for history. |
Papers 21 · 4 marked discovery
| D-CIPHER ↗ discovery | A multi-agent system with a planner and different executor agents for offensive security tasks. |
|---|---|
| Hackers or Hallucinators? ↗ discovery | An analysis of LLM-based automated penetration testing. |
| Incalmo ↗ discovery | An LLM-assisted system for red teaming multi-host networks. |
| PentestAgent ↗ discovery | LLM agents that automate penetration testing steps from intelligence gathering to exploitation. |
| A Survey of LLM-Driven Penetration Testing ↗ | A survey that sorts LLM pentest agents into a taxonomy and lists the open challenges. |
| Agentic Security: Tools, Failure Modes and Design Laws ↗ | A systematization of LLM-driven pentest tools, how they fail and what design rules follow. |
| Artificial Intelligence as the New Hacker ↗ | An early paper on developing AI agents for offensive security. |
| AutoPT ↗ | Asks how far LLM agents are from end-to-end automated web penetration testing. |
| Benchmarking Practices in LLM-driven Offensive Security ↗ | A review of the testbeds, metrics and experiment design used to benchmark offensive LLM agents. |
| Big Enough to Break Out ↗ | Tracks how the pentest capability of LLM agents has risen across model generations. |
| CAI: An Open, Bug Bounty-Ready Cybersecurity AI ↗ | Describes CAI, an open framework for cybersecurity AI aimed at bug bounty work. |
| Co-RedTeam ↗ | Multi-agent LLM system for security discovery and exploitation. |
| From Controlled to the Wild ↗ | Evaluates pentesting agents beyond controlled benchmarks and on real-world targets; its benchmark code is EthiBench. |
| HackSynth ↗ | An LLM agent and evaluation framework for autonomous penetration testing. |
| On the Surprising Efficacy of LLMs for Penetration-Testing ↗ | A review of LLM pentesting research and where it falls short. |
| PenHeal ↗ | A two-stage LLM framework that automates pentesting and then recommends fixes. |
| PentestGPT ↗ | The paper behind PentestGPT, an LLM-driven automatic penetration testing tool. |
| Systematic Capability Benchmarking of Frontier LLMs for Offensive Cyber Tasks ↗ | Benchmarks frontier LLMs on offensive cyber tasks. |
| The Ethics of Autonomous AI Agents for Offensive Security ↗ | Discusses the ethical questions in building and releasing autonomous offensive agents. |
| VulnBot ↗ | A multi-agent collaborative framework for autonomous penetration testing. |
| xOffense ↗ | A multi-agent pentest framework built on domain-adapted LLMs. |
Trackers 5
| AI Hacking Incidents ↗ | Sourced cyberattacks since 2023 where attackers used AI, with which harness, what the AI did and the primary sources. |
|---|---|
| AI Incident Database ↗ | Public database of reported AI harms and misuse incidents, searchable by system and date. |
| awesome-ai-agent-attacks (webpro255) ↗ | A timeline of real AI agent security incidents and vulnerabilities from 2024 to 2026, each entry sourced and dated. |
| MITRE ATLAS ↗ | Knowledge base of adversary tactics against and with AI systems, with real-world case studies. |
| OECD AI Incidents and Hazards Monitor ↗ | The OECD's monitor of AI incidents and hazards, with its methodology. |
Threat report series 8
| Anthropic: Detecting and countering misuse of AI, August 2025 ↗ | Anthropic's threat intelligence report on AI cybercrime and other abuses of Claude. |
|---|---|
| Anthropic: Disrupting an AI-orchestrated cyber espionage campaign ↗ | Anthropic's report on a highly sophisticated AI-led cyberattack and how it was disrupted. |
| CrowdStrike Global Threat Report ↗ | CrowdStrike's annual report on rising attacks and adversary tactics. |
| Google Threat Intelligence blog ↗ | GTIG and Mandiant reports on adversary use of AI, the most frequent primary source in the attack table. |
| GTIG AI Threat Tracker ↗ | Google's findings on adversarial misuse of AI, including Gemini and non-Google tools. |
| GTIG: Adversarial Misuse of Generative AI ↗ | Google's findings on government-backed and information-operations actors using the Gemini web app. |
| Microsoft Security Insider intelligence reports ↗ | Microsoft's threat intelligence reports on nation-state actors, cybercrime and other threat actors. |
| Unit 42 research ↗ | Palo Alto Networks incident and threat research, source of the Hermes Agent and Cairn cases. |
04 // THE DATASETTake it with you
Free to use
Cite the primary reports
JSONdata.json
Every harness, use tag, quoted sentence, source and date on the site, in one file. The page renders from the same data, so the two cannot disagree.
SUBSTRATEDataset mirror
The dataset as JSON and CSV in Substrate, the open repository of data and research behind our projects, with a summary and an update log.
RESEARCHNotes and method
The full research notes, including what the page leaves out: commercial coding agents used as harnesses, Hugging Face as infrastructure, dark-LLM services and the hype list.
05 // SISTER SITEAI Hacking Incidents
The attacks, not the tools
Incidents checked 2026-11-17
LEDGERhackingincidents.ai
A sourced ledger of cyberattacks where AI was known or strongly suspected to be used, from 2023 on, each graded and linked to its sources. This index asks which open-source harnesses attackers used; the ledger asks what happened in each attack. Where they meet, each links the other: 9 harnesses here link 12 incidents there, and the ledger links its incidents back to the rows here.
SYNCKept in step
Each site reads the other's published data when it refreshes, and refreshing one refreshes the other, so a new incident that names a harness, or a new harness an incident names, shows up on both. When the ledger names an open-source harness this index does not track, the refresh lists it for a check against the roster rule.
06 // LIMITSWhat it can't tell you
Read the table with these in mind
No record is not no useA harness without an attack row has none in the primary reporting we have found. Attackers rarely announce their tooling, and most intrusions are never written up.
Open source onlyCommercial coding agents such as Claude Code, Codex and Gemini CLI show up in more of the attack reports we have read than any open-source harness, but they are a different question and stay in the research notes.
Tags are claimsOutlined tags say what a project's README says it does. Nobody here ran the tools to confirm it. Filled tags are what a report saw.
Reports hedgeWhen a report says "likely" or names a harness once in passing, the row says so and the tier stays at Observed.
Built by Daniel Miessler, an AI and cybersecurity researcher and founder in the San Francisco Bay Area who has been building and writing online since 1999. His work helps people and companies articulate and pursue their Ideal State.
danielmiessler.com ↗