What an AI security assessment checks

Discovery uses the records an AI tool leaves in systems the business already runs. Microsoft Entra ID holds a permissions record for every enterprise application. That record splits what an administrator consented to on behalf of the organisation from what individual users consented to for themselves. Its audit log carries a Consent to application entry each time a user grants an app access. Those records catch AI plugins connected through OAuth instead of installed by IT. Network and DNS logs show AI services being reached that appear in no register. Asking team leads what they have bought catches tools charged to a card and recorded nowhere central.

Each tool found is then sorted by how it arrived. A tool built by the business, a tool bought under contract and a tool an individual signed up to carry different approvals. Each carries different contractual terms and different people able to change them. The assessment records which of those approvals exist and which were never given.

Once a tool is on the list, the assessment establishes what it can read, write to and retain. Engaging with artificial intelligence, published in January 2024 by the Australian Signals Directorate's Australian Cyber Security Centre with international partner agencies, sets the questions to put to each tool. It asks how the AI system collects, processes and stores data, and whether it can meet the organisation's data residency or sovereignty obligations. AI systems are often hosted in the cloud and may send data between different regions. For a third-party system the same guidance says to understand whether the organisation's inputs will be used to retrain the model. It also says to consider a private version of the system where one is available. The same guidance says to establish how the organisation's data will be managed if the commercial agreement with the vendor ends.

Where that data is personal information, the Privacy Act 1988 already applies, and adopting an AI tool does not change which entities it covers. The Act covers an Australian business whose annual turnover has been more than $3 million in any financial year since 2002, so a business whose turnover has since dropped under that figure stays covered. The OAIC also lists categories covered whatever the turnover, among them health service providers, businesses that trade in personal information, credit reporting bodies, contractors under a Commonwealth contract, businesses accredited under the Consumer Data Right and reporting entities under the Anti-Money Laundering and Counter-Terrorism Financing Act 2006. The OAIC's guidance on privacy and the use of commercially available AI products states that privacy obligations apply to personal information input into an AI system. They also apply to output data generated by AI where that output contains personal information. Australian Privacy Principle 6 limits use or disclosure of that information to the primary purpose it was collected for, unless the individual consents or an exception applies. The OAIC lists as a common exception a secondary use the individual would reasonably expect, where that purpose is related to the primary purpose, or directly related where the information is sensitive. Further exceptions in the Act include a use or disclosure required or authorised by or under an Australian law or a court or tribunal order.

The same guidance sets an expectation on product selection. Due diligence covers whether the product has been tested for its intended use and how human oversight can be embedded into the process. It also covers the privacy and security risks, and who will have access to the personal information put into the product or generated by it. The OAIC states that due diligence for AI products should not amount to a set and forget approach. It recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available AI chatbots and other publicly available generative AI tools.

Australia's National AI Centre published the Voluntary AI Safety Standard on 5 September 2024 through the Department of Industry, Science and Resources. It sets out 10 voluntary guardrails that apply to all organisations throughout the AI supply chain. Guardrail 2 requires a risk management process to identify and mitigate risks, and states that AI impact and risk management processes need to consider how the AI system is used. Guardrail 3 requires organisations to protect AI systems and implement data governance measures to manage data quality and provenance. Guardrail 9 requires records to be kept and maintained that allow third parties to assess compliance with the guardrails. It names an AI inventory and consistent AI system documentation as part of those records.

The Centre published Guidance for AI adoption on 21 October 2025. It describes the guidance as updated and simplified, evolving the Voluntary AI Safety Standard and built around 6 essential practices. Its implementation version asks organisations to maintain an AI register: an up to date, organisation-wide inventory of each AI model and system. That inventory is held with enough detail to support future conformance assessments. The register records the accountable people, the purpose and business goals, and the capabilities and limitations of the model or system. It also records the system's origin and any fine-tuning, and the datasets used for training and testing and where they came from. It records the identified risks, the risk treatment plan, and the dates of review.

Naming an access boundary in a policy does not enforce it. The ACSC guidance recommends phishing-resistant multi-factor authentication, for example FIDO2 security keys, for access to an organisation's AI systems including any repositories that hold training data. It recommends granting privileges on the need-to-know principle and the principle of least privilege, and limiting the number of accounts with access to the AI's development and production environments. It also requires that privileged accounts are routinely revalidated and disabled after a set period of inactivity. The assessment tests those settings on each boundary it records.

An assessment answers what is in use, what it can reach and what policy governs it. A penetration test answers whether the system holds when someone works against it. That work is manual testing against the OWASP Top 10 for Large Language Model Applications, and prompt injection and jailbreak attempts run against the configuration the application is actually running. Findings are reproduced and rated for severity before they reach a report.

A test run before the boundaries are set reports on a configuration that has moved by the time the fixes land. A retest scoped as a confirmation becomes the first measurement of the configuration now running.

The National Institute of Standards and Technology released its AI Risk Management Framework on 26 January 2023. Its core has four functions: Govern, Map, Measure and Manage. Govern applies to all stages of an organisation's AI risk management processes and procedures, and the framework describes it as a cross-cutting function infused throughout AI risk management. Map, Measure and Manage can be applied in AI system-specific contexts and at specific stages of the AI lifecycle. The framework states that its actions are not a checklist and not necessarily an ordered set of steps. Once the Govern outcomes are instituted, most users would start with Map and continue to Measure or Manage, iterating between functions as needed.

An assessment produces the material Map works from: the context each system operates in, the data it touches and the people answerable for it. The policy and the access boundaries it sets are what Govern holds afterwards. Guardrail 1 of the Voluntary AI Safety Standard puts a name against that responsibility. It states that leaders cannot delegate or outsource accountability for the safe and responsible deployment and use of AI systems. The first of the 6 essential practices in Guidance for AI adoption is the same instruction: decide who is accountable. The AI register carries that person against each system.

The order of the steps decides whether the output holds. A usage policy written against a partial inventory governs only the tools someone thought to declare, and an access boundary set before data reach is mapped has not been checked against what the tool can actually reach.

Black Shard delivers the assessment and the penetration test that follows it as separate AI cyber defence engagements. The boundaries recorded by the first are what the second is scoped against.