Skip to content

AI penetration testing, explained

AI penetration testing means two things: penetration testing of AI systems, such as LLM features and agents, and penetration testing done by AI. This guide explains both for teams that ship AI, and shows why your product needs them in the same test.

What is AI penetration testing?

AI penetration testing is a term with two meanings. It can mean attacking AI systems, such as chat features, RAG pipelines and agents, to find where they break. It can also mean using AI to do the attacking: autonomous testing that finds and exploits vulnerabilities in any software. Buyers often mix them up.

The confusion costs money. One team needs its LLM feature tested and buys an autonomous scanner pointed at its login page. Another wants faster web testing and sits through a pitch on model red teaming. Same phrase, different product.

The quick check: ask what the target is and who the tester is. The target tells you whether it is testing of AI. The tester tells you whether it is testing by AI. The two combine, and for products that ship AI they belong together.

What is penetration testing of AI systems?

Penetration testing of AI systems attacks the AI parts of your product: the chat interface, the retrieval pipeline, the system prompt and any agent that can call tools. The goal is to make the model leak data, ignore its instructions or act on an attacker's behalf, and to prove it happened.

These categories come from the OWASP Top 10 for LLM Applications 2025 and the OWASP Top 10 for Agentic Applications for 2026. A good test names the category for each finding, so your team knows what class of problem it is fixing.

  • LLM01 Prompt Injection: a message, web page or uploaded file carries instructions the model follows. The assistant stops serving your user and starts serving the attacker.
  • LLM02 Sensitive Information Disclosure: the model reveals another customer's records or internal documents pulled in through retrieval.
  • LLM07 System Prompt Leakage: the hidden instructions come out, along with any keys or business rules written into them.
  • LLM05 Improper Output Handling: model output lands in a page or a query unescaped and becomes script injection or a database attack.
  • ASI01 Agent Goal Hijack and ASI02 Tool Misuse: an agent is steered into using its tools, such as email, refunds or file access, for the attacker's goal.
  • ASI06 Memory & Context Poisoning: content planted today changes what the agent does for other users tomorrow.

What is penetration testing done by AI?

Penetration testing done by AI is autonomous testing: software that maps a target, chooses attacks, exploits what it finds and writes up the result, with no wait for a human tester's calendar. The target can be a web app, an API or an AI feature. The gain is speed, coverage and frequency.

Speed changes what testing is for. A traditional pentest is an annual event. An autonomous test can run every week, against the release you shipped on Tuesday, and tell you what is new and what got fixed since the last run.

The known weakness is proof. In a Pentest-Tools survey of 158 respondents in June 2026, 87.8% hit AI pentest findings that needed significant manual validation, and 53% ranked proof of exploit as a top buying factor. An autonomous finding is worth what its evidence is worth.

That is why Grapefruit attaches the exact request and response to every finding. Your engineer pastes the request, runs it and watches the attack work.

Testing of AI vs testing by AI: how do they compare?

Testing of AI describes the target. Testing by AI describes the tester. One finds weaknesses specific to models and agents, such as prompt injection and tool misuse. The other finds any exploitable weakness faster and more often. The table sets them side by side, with the questions to ask a vendor.

Neither column is complete alone. One has depth on AI and no reach into the application. The other has reach and speed, and needs AI coverage added. A product that ships AI needs the full row.

Penetration testing of AIPenetration testing by AI
What it isAttacking LLM features, RAG and agentsAutonomous software doing the attacking
What gets testedYour chat, retrieval, prompts and agent toolsAny target: web app, API or AI feature
What it findsPrompt injection, data leakage, system prompt leakage, tool misuse, goal hijackBroken access control, injection and authentication flaws, plus AI issues when the tester covers them
Standards it maps toOWASP Top 10 for LLM Applications 2025, OWASP Top 10 for Agentic Applications for 2026, MITRE ATLASOWASP Web Security Testing Guide (WSTG) v4.2, OWASP API Security Top 10 2023
Main benefitCovers the part of your product most tests skipResults in hours and re-tests every week
What to ask a vendorDo you test agents and tools, or just the chat box? Is the app behind the AI in scope?Is every finding exploited and proven? Can I replay the request myself?

Why does a product that ships AI need both at once?

Because real attacks chain across the layers. An attacker does not stop at the chat box. They use the model to reach the API, the API to reach the data, and the agent's tools to act. Testing the AI and the application separately leaves the chain between them untested.

Here is how a chain plays out. An attacker hides instructions in a support ticket. Your AI agent reads the ticket, follows the instructions and calls an internal API to export customer records. The API allows it, because the agent's service account can read every tenant. Three small weaknesses, one breach: prompt injection, excessive agency and broken object level authorization.

Most vendors sell AI red teaming and app pentesting as separate engagements with separate reports. Grapefruit attacks the web app, the APIs, the LLM features and the agents in one test, and reports them in one place with the requests that prove each finding.

The fix rate shows how much help teams need here. Cobalt's 2025 State of Pentesting data, as reported by CSO Online, shows 48% of pentest findings get fixed, 69% of serious ones, and only 21% of findings in generative AI apps. A finding your engineer can replay is far easier to fix.

Which standards apply to AI penetration testing?

Four references anchor AI penetration testing: the OWASP Top 10 for LLM Applications 2025, the OWASP Top 10 for Agentic Applications for 2026, MITRE ATLAS, and the NIST AI Risk Management Framework 1.0 with its Generative AI Profile. Together they name the risks, the attacker techniques and the governance that testing supports.

For the application underneath, the OWASP Web Security Testing Guide (WSTG) v4.2 and the OWASP API Security Top 10 2023 still apply. Your AI feature runs on a web app and an API, and attackers use all of it.

  • OWASP Top 10 for LLM Applications 2025: the ten main risk categories for LLM features, from LLM01 Prompt Injection to LLM10 Unbounded Consumption.
  • OWASP Top 10 for Agentic Applications for 2026: the risks that appear when a model can act, from ASI01 Agent Goal Hijack to ASI10 Rogue Agents.
  • MITRE ATLAS: a knowledge base of adversary tactics and techniques against AI systems, useful for describing how an attack unfolded.
  • NIST AI Risk Management Framework 1.0 (NIST AI 100-1): a framework organized around four functions, govern, map, measure and manage. Testing gives the measure function real data.
  • Generative AI Profile (NIST AI 600-1): the companion profile that applies the framework to generative AI.

What should you ask an AI penetration testing vendor?

Ask what is tested, how findings are proven and how often the test repeats. A vendor that covers just the chat box, delivers findings without a working exploit, or tests once a year leaves gaps in the places where a product that ships AI changes fastest.

Then ask for a sample report. Find one AI finding in it and check whether you could reproduce the attack from what is on the page.

  • Do you test agents and their tools, or just prompts?
  • Are my web app and APIs in the same test and the same report?
  • Does every finding include the request and response that proves it?
  • Do you test behind the login, with accounts I supply?
  • How soon do results arrive after kickoff?
  • What happens after I fix something? Is it re-tested?
  • Which OWASP lists and frameworks does the testing follow?

How does Grapefruit run AI penetration testing?

Grapefruit is both meanings in one product: autonomous testing with real exploitation, aimed at your web app, your REST and GraphQL APIs, your LLM features and your agents together. Every finding is ranked critical, high, medium or low and arrives with the captured request and response that proves it.

Setup takes a few minutes. Add your hostname, publish one DNS TXT record and sign the testing authorization online. Supply test accounts for authenticated coverage. Results land in your portal within hours of kickoff.

Choose a full penetration test at a fixed quote, or a maintenance plan that re-tests your verified domains every week or month. Each run is dated and shows what is new and what got fixed. See pricing, or open the sample report to see a finding you can replay.

Questions

What does AI penetration testing mean?
It means one of two things. Either penetration testing of AI systems, where LLM features, RAG pipelines and agents are the target, or penetration testing done by AI, where autonomous software performs the attack on any target. Ask a vendor which one they sell. Products that ship AI need both in one test.
Is AI penetration testing the same as AI red teaming?
They overlap. AI red teaming usually focuses on the model and its behavior: jailbreaks, harmful output, prompt injection. AI penetration testing of a product goes further into the system around the model, including the APIs it calls, the data it retrieves and the tools its agents use, and proves impact with a working exploit.
Can AI find real vulnerabilities, or just guesses?
Autonomous testing finds real, exploitable vulnerabilities when it actually exploits them and captures the evidence. Proof is the difference. In a Pentest-Tools survey from June 2026, 87.8% of respondents hit AI findings needing significant manual validation. Grapefruit attaches the exact request and response to every finding so you can replay it.
Do I need AI penetration testing if I only use a third-party model API?
Yes. The model provider secures the model. Your prompts, your retrieval data, your agent tools and the APIs behind them are yours. Most findings in LLM features, such as prompt injection that reaches customer data or an agent misusing a tool, live in the application you built around the model.
Which frameworks should an AI penetration test follow?
Look for the OWASP Top 10 for LLM Applications 2025 for LLM features, the OWASP Top 10 for Agentic Applications for 2026 for agents, and MITRE ATLAS for adversary techniques. For governance, test results support the NIST AI Risk Management Framework 1.0 and its Generative AI Profile.
How often should AI features be penetration tested?
As often as they change. Prompts, models, retrieval sources and agent tools change far more often than a yearly test can follow. A weekly or monthly re-test checks each release, and a dated record of every run shows what is new and what got fixed since the last one.

Standards and sources