An AI penetration test targets the model and the tools it can reach. A standard test targets everything underneath.
A business running a live AI feature usually needs both: a standard test of the application, network and identity layer underneath, and an AI test of the prompt injection, guardrail and retrieval surface the model adds.
A standard penetration test and an AI penetration test answer different questions about the same system. The standard test asks whether your network, applications, APIs and identity paths hold up under manual testing. The AI test asks whether the model on top of that system can be redirected, made to disclose data it should not, or pushed to misuse the tools it has been given.
Black Shard runs both as fixed-scope engagements under the same written scope and rules of engagement. A business that ships a chatbot, copilot or agent usually scopes both.
Side by side.
| Criteria | AI penetration testing | Standard penetration testing |
|---|---|---|
| What is being tested | Model behaviour, prompt handling, guardrails, retrieval permissions, and what an agent can do with the tools it has been given. | Network, application and API attack surface, identity and access paths, and cloud configuration. |
| Methodology | The OWASP Top 10 for Large Language Model Applications 2025 for the model surface, the OWASP Top 10 for Agentic Applications where the feature can call tools, and MITRE ATLAS for the techniques catalogued against AI-enabled systems. | OWASP guidance for the application and API, ASD guidance for networks, and identity, exposure and configuration review for cloud environments. |
| Access provisioned | What a normal user of the feature has, plus the configuration behind it where the scope calls for it: the system prompt, guardrail settings, retrieval sources or tool permissions. | Credentials, network ranges, API endpoints and environments named in the written scope. |
| How a finding is confirmed | Repeated attempts against each technique. OWASP notes that the stochastic behaviour of models leaves it unclear whether prompt injection can be fully prevented, so one failed attempt does not clear a guardrail. | A single reproduction is normally enough, because the system does not vary its answer to the same input. |
| Where the fix lands | The system prompt, the guardrail configuration, the retrieval permission model, or the application wrapped around the model. | Application code, network configuration, patching, or identity and access settings. |
| What it does not cover | Authentication, session handling and access control on the application the AI feature sits inside. That layer is still tested the standard way. | The model surface. Standard testing does not probe prompt injection, jailbreak attempts, or data leakage through retrieval. |
| What triggers the next test | A change to the AI feature: a new model, a new tool connection, a new retrieval source, or a changed system prompt or guardrail. | ASD control ISM-2118 sets the baseline: vulnerability assessments and penetration tests before deployment, before a significant change, and at least every six months after. |
When you need an AI penetration test
- You operate a chatbot, copilot or agent that answers from a knowledge base, calls internal tools, or acts on another system.
- A retrieval pipeline draws on client or internal data, and no one has tested whether it can surface data outside the requester's access.
- An agent has been given access to internal tools, and its permission boundary has never been tested.
- A buyer or auditor has moved from asking for a penetration test to naming the AI feature or prompt injection.
When a standard penetration test is enough
- The system in scope has no AI feature: it is the network, the application, the API or the cloud environment.
- You are testing the application an AI feature sits inside, and the model surface is scoped as its own engagement.
- Your testing cadence covers infrastructure and application risk, and no chatbot, copilot or agent has shipped since the last test.
- A procurement or audit request asks for penetration testing without naming an AI feature.
What an AI feature adds to the attack surface
Standard testing already covers the layer an AI feature sits on: authentication, session handling, access control, injection and application logic against OWASP guidance, plus networks and cloud environments against ASD guidance. An AI feature adds a layer none of that reaches. Natural language becomes an input an attacker can use to redirect the system's behaviour, and the model's output is trusted downstream in ways a fixed API response is not.
The OWASP Top 10 for Large Language Model Applications 2025 names the categories an AI penetration test works through: prompt injection at LLM01, sensitive information disclosure at LLM02, improper output handling at LLM05, excessive agency at LLM06, and vector and embedding weaknesses at LLM08, which covers unauthorised access and cross-context leakage where users or applications share a vector database. OWASP treats jailbreaking as a form of prompt injection, and separates direct injection, typed into the input, from indirect injection, planted in content the system later retrieves.
OWASP's mitigation list for prompt injection includes regular penetration testing that treats the model as an untrusted user, to test the effectiveness of trust boundaries and access controls. Where the feature can call tools, the OWASP Top 10 for Agentic Applications separates tool misuse from identity and privilege abuse.
Why the two tests confirm findings differently
A standard test reproduces a finding once. If a request returns another tenant's data, the system returns it again on the next attempt, and one reproduction is enough to write it up.
Model behaviour is probabilistic. An injection or jailbreak attempt that fails once can succeed on a later attempt against the same input, so an AI penetration test runs repeated attempts against each technique before treating a guardrail as holding. A re-test after a fix runs the same way, with repeated attempts against each fixed technique.
Where the two engagements overlap
A chatbot, copilot or agent runs on top of the same login, session and access control layer as the rest of the application, and that layer does not stop needing a standard test because a model now sits on it. Testing the AI feature alone misses the account takeover or tenancy breach that starts underneath it.
The reverse also holds. A standard test run against an application with a chatbot attached assesses the login form, the API and the infrastructure, and stops there. The two engagements are scoped separately.
Scoping and delivery
Both engagements start with a written scope and quote, rules of engagement agreed before testing begins, and a re-test of agreed fixes limited to the findings in the original report. An AI test adds to that scope the AI features and integration points involved, whether retrieval or tool access is in play, and the content the system is excluded from generating even to prove a finding.
Black Shard builds and operates production AI systems as well as testing them. Aurii, clinical software Black Shard built and operates, runs speech recognition and document pipelines against clinical dictation in a multi-tenant Postgres database enforced with row-level security in Azure's Australia East region. Findings are held in confidence, and disclosure to a third party, including a model provider, is made only with the client's consent.
Questions, answered
- Does an AI penetration test replace a standard penetration test?
- No. An AI feature sits on top of the same authentication, session handling and access control as the rest of the application, and that layer is still tested the standard way. The AI test adds the surface a standard test does not reach: prompt injection, jailbreak attempts, improper output handling and data leakage through retrieval.
- What is prompt injection?
- An input that alters a model's behaviour in a way the system was not designed to allow. OWASP separates direct injection, typed into the prompt, from indirect injection, planted in a document, email or web page the system later retrieves and treats as trustworthy. ASD describes it as an input manipulation attack that can hijack a model's output and jailbreak the system.
- We already use Microsoft Copilot. Do we need an AI penetration test?
- Microsoft documents Copilot as surfacing only organisational data the individual user already has at least view permissions to, and states that overshared content across SharePoint, OneDrive and Teams affects what Copilot returns. The work that answers that is a Copilot security review of those permission boundaries. An AI penetration test applies to an AI feature you build, host or configure.
- How does a re-test work for an AI penetration test?
- The same as a standard re-test, confined to the findings in the original report. Where a standard re-test confirms a fix once, an AI re-test runs repeated attempts against each fixed technique before treating the guardrail as holding.
- Which one does an auditor or security questionnaire expect?
- A request for a penetration test report, without more, is answered by the standard engagement. Where the request names an AI feature, prompt injection, a model or a copilot, it is asking for the AI test, and a standard report will not close it.
Send what the AI feature does and what it can reach.
Brisbane head office. Work delivered across Australia.