Guardrails and audit trails before an AI feature ships

A guardrail is a check that runs in code. On the way in, it treats anything a user can influence, a message, an uploaded file, a retrieved document, as untrusted input, the same discipline applied to an API payload. On the way out, it validates the response before that response is rendered into a page, written to a database, or passed to another system as an instruction.

The OWASP Top 10 for Large Language Model Applications, 2025 edition, names the failure modes a guardrail has to close. Prompt injection, LLM01, occurs when user prompts alter the model's behaviour or output in unintended ways, and OWASP records that retrieval augmentation and fine-tuning do not fully mitigate it. Improper output handling, LLM05, is insufficient validation, sanitisation and handling of model output before it passes downstream to other components and systems. OWASP lists cross-site scripting, server-side request forgery, privilege escalation and remote code execution among the results. Excessive agency, LLM06, has three root causes OWASP names directly: excessive functionality, excessive permissions and excessive autonomy.

System prompt leakage, LLM07, is commonly read as the risk that the hidden instruction gets extracted. OWASP takes a harder line. The system prompt should not be treated as a secret, and should not be used as a security control. The risk sits in an application that delegates session management and authorisation checks to the model, or that keeps credentials, connection strings or role definitions in prompt text.

The record has to answer three questions after the fact: what did the feature read, what did it return, and what happened next. That means logging the input the model received, including anything retrieved from a document store, and the response it produced. It also means logging any action taken on the strength of that response: a tool call, a database write, or a message sent. Log that action against a timestamp and the account that triggered it.

ASD's Australian Cyber Security Centre published Engaging with artificial intelligence in January 2024 with partner agencies from ten other countries. It asks organisations to log and monitor outputs to detect a change in behaviour or performance that may indicate a compromise or data drift. It also asks organisations to log and monitor inputs to meet compliance obligations and aid investigation after an incident. It asks organisations to log and monitor for high frequency, repetitive prompts, a sign of automated prompt injection attacks. It also asks organisations to establish a baseline of the AI system's activity, so they can determine when logged events are anomalous.

Australian Privacy Principle 11 under the Privacy Act 1988 requires an entity to take reasonable steps to protect the personal information it holds from misuse, interference and loss. The same principle covers unauthorised access, modification or disclosure. The OAIC's guidance on privacy and the use of commercially available AI products, published 21 October 2024, applies that obligation to AI specifically. Where a product will be integrated into the entity's systems and have access to its documents, it asks whether that happens within a secure environment on the entity's premises or hosted through the cloud.

The log has to exist independently of the feature because the behaviour is not fully repeatable. When a customer reports that the feature surfaced something it should not have, the answer comes from the record of that interaction. Re-running the feature produces a second interaction, and no evidence about the first.

The boundary between one customer's data and another's belongs in code that runs before the model sees a token. An instruction telling the model to answer only from one customer's records travels on the same input channel that carries prompt injection. OWASP is explicit that the system prompt is not a security control.

OWASP's vector and embedding weaknesses category, LLM08, describes the failure in a retrieval feature. In multi-tenant environments where multiple classes of users or applications share the same vector database, there is a risk of context leakage between users or queries. The mitigations OWASP gives are fine-grained access controls, permission-aware vector and embedding stores, and strict logical and access partitioning of datasets in the vector database. Embedding inversion, recovering source information from the embeddings themselves, sits in the same category.

The NIST Generative AI Profile, AI 600-1, published July 2024, carries twelve risk categories. Data privacy covers impacts from leakage and unauthorised use, disclosure or de-anonymisation of personally identifiable or sensitive information. Information security covers the expanded attack surface of a generative system, and names prompt injection and data poisoning among the attacks against it.

The working control is retrieval scoped by tenant identity at the data layer, the same discipline as row-level isolation elsewhere in the stack. Before release, attempt a genuine cross-tenant read against the feature and confirm the attempt fails.

A reviewer has four extra questions beyond the correctness of the code. What data reaches the prompt, and does it carry more than the task needs. What is the model authorised to call, and is that permission scoped to one action or to a whole account. How is the output consumed, since rendering it as text is a different risk to executing it or passing it as an instruction onward. What does a wrong answer look like, given a plausible wrong answer passes any test that only checks the response is well formed.

OWASP's mitigations for excessive agency read as review checkpoints. Limit the functions an extension implements to the minimum the task needs, and limit the permissions it holds on downstream systems. Track user authorisation and security scope so actions taken on a user's behalf run in the context of that specific user, with the minimum privileges necessary. Implement authorisation in downstream systems instead of relying on the model to decide whether an action is allowed. The ACSC guidance sets the same boundary at the account level: privileges on need-to-know and least privilege. It also holds down the number of accounts reaching development and production environments and training data repositories.

System instructions set how the feature behaves and are not covered by the compiler or the test suite. Version them, review a change before it ships, and keep production edits inside that path. The Guidance for AI Adoption asks for an AI register recording each system's purpose, capabilities, limitations, acceptance criteria, test results and dates of review. An instruction edited in production outside version control leaves that register describing a system that no longer exists.

The test and monitor practice in the Guidance for AI Adoption asks an organisation to define and communicate clear acceptance criteria and test methodologies. Those criteria have to reflect the intended use, context and potential risks. It asks the organisation to conduct testing before deployment, and to document tests and outcomes so they support external audit and oversight. It also asks for documented deployment authorisation and rationale from the accountable person, based on the test results. For developers of general-purpose AI systems it names safety evaluations that scale with model capabilities, including testing for jailbreaking or prompt manipulation and comprehensive red teaming to identify vulnerabilities.

NIST places the same work in the measure function, one of four in the AI Risk Management Framework alongside govern, map and manage, released January 2023. Under MEASURE 2.7, which covers AI system security and resilience, the Generative AI Profile asks organisations to perform AI red-teaming. The targets include prompt injection, adversarial examples, data poisoning, membership inference and model extraction.

A test that only confirms correct answers on the cases the builders thought of does not meet that bar. The test report has to name what was attempted, which attempts succeeded, and what was fixed before the deployment authorisation was signed.

Run these in order. An audit log becomes evidence once the guardrails it records are enforced in code.

Black Shard runs the manual testing in the last step: prompt injection, jailbreak and guardrail bypass, insecure output handling, tool and agent permission abuse, and cross-tenant data leakage. It runs under the same written scope and rules of engagement as any other engagement. It sits in the AI cyber defence service. We run the same checks on aurii, the AI medical workflow platform we build and operate on Azure for Australian private hospital specialists. We run them there before we run them on a client's feature.