How a prompt becomes a disclosure under the Privacy Act
AI data leakage under the Privacy Act happens when a prompt to a commercial AI tool carries personal information outside the business. In Australia that is either a use or a disclosure under the Privacy Act 1988. A business that types a customer name or a case file into a commercial AI product is handling personal information under that Act. The OAIC's Guidance on privacy and the use of commercially available AI products, published in October 2024, draws the line at control. Entering information into an AI system is a use where the data stays within the organisation's control. It is a disclosure where the data is made accessible to others outside the organisation and has been released from the organisation's effective control. Both have to comply with APP 6.
Which side of that line a prompt falls on is set by the product's terms and settings. The guidance directs a business to check whether the service terms give the developer access to data the business inputs or the product generates. It also directs a check on whether any feature passes that data to a third party, naming interfaces to search engines as one case. Where a proprietary system carries protections that stop information entered into it being disclosed outside the organisation, including to the system developer, the OAIC treats that as a use. It is not a disclosure. Where the terms give the developer access to the inputs, staff entering a customer's details are disclosing them to the owners of the product. The guidance's worked example states the position without that condition. Employees at an insurance company enter a customer's claim, including the customer's sensitive health information, into a publicly available chatbot, and by entering that information the company is disclosing it to the owners of the chatbot.
APP 6 is not the only principle on an AI input. The same guidance names the generation or collection of personal information under APP 3, accuracy under APP 10 and cross-border disclosure under APP 8 as obligations to consider alongside it. A business training or fine-tuning a model on its own data sits under a second OAIC guidance document issued the same day, with a different set of duties.
What APP 6 requires before a new purpose is added
APP 6 allows personal information to be used or disclosed for the purpose it was collected for, the primary purpose, or for a secondary purpose where an exception applies. The two exceptions a business reaches for are consent, and the reasonable expectations test. That test asks whether the individual would reasonably expect the use or disclosure for the secondary purpose. It also asks whether that purpose is related to the primary purpose, or directly related where the information is sensitive.
The OAIC sets a high bar on the second of those for anything AI-related. Its guidance states that in many cases it will be difficult to establish that a secondary use for an AI-related purpose was within reasonable expectations. Training an AI system is its example. Where a business cannot clearly establish that the secondary use was within reasonable expectations and related to the primary purpose, the guidance says it should seek consent for that use. It should also offer individuals a meaningful and informed ability to opt out, or do both, to avoid regulatory risk. It also expects the information entered to be the minimum amount sufficient for the secondary purpose.
The assessment runs against the purpose the information was collected for, so a single approval covering a product will not carry every use of it. The OAIC's worked example splits one product two ways. Running customer data through an AI system to produce tailored marketing recommendations is likely consistent with the primary purpose where that was one of the purposes of collection. Using the same customer data to fine-tune the AI system is a further purpose that needs consent or an established reasonable expectation. Where a purpose is ambiguous, the OAIC construes it narrowly rather than expansively.
What the OAIC expects before a commercial AI product is adopted
The same guidance sets the due diligence expected before a commercially available product is adopted. A business should consider whether the product has been tested for its intended uses and how human oversight can be embedded into the processes around it. It should also weigh the potential privacy and security risks, and who will have access to the personal information input into the product or generated by it. The guidance adds that due diligence should not amount to a set and forget approach. Review of the product's performance, staff training and monitoring should run across the product lifecycle.
Where the product requires personal information to be disclosed to the developer or to a third party, the guidance sets out the measures to weigh. These are turning off the features that would make that disclosure, giving the individual any required notice of it, and putting controls in place that stop personal information being entered. A further option is prohibiting staff from entering personal information in AI inputs at all. It states that any such control or prohibition needs training and auditing behind it. As a matter of best practice, the OAIC recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available AI chatbots and other publicly available generative AI tools.
None of that review reaches a tool the business does not know about. A staff member who signs up for a free AI product with a work email has made the access decision alone. The due diligence has nothing to attach to until the tools in use are recorded. The OAIC pairs the due diligence with a privacy impact assessment. It states that assessment will assist an organisation to understand the privacy impact of a particular AI product. It also helps identify ways to manage, minimise or eliminate that impact.
What changes when the business trains or fine-tunes a model itself
Training or fine-tuning a generative AI model on your own data falls under the OAIC's Guidance on privacy and developing and training generative AI models. That guidance focuses on APP 1, APP 3, APP 5, APP 6 and APP 10, in the context of planning generative AI and compiling a dataset for training. A developer under that guidance includes any organisation that designs, builds, trains, adapts or combines AI models and applications. The guidance names adapting a trained model through fine-tuning as one of the ways an organisation becomes a developer. A business fine-tuning a hosted model on its own support tickets is inside the definition.
Collection has to be lawful and fair under APP 3. The guidance states that it will generally be unfair to collect personal information covertly without the knowledge of the individual, although that depends on the circumstances. Creating a dataset through web scraping may itself be a covert and therefore unfair means of collection. Sensitive information inadvertently collected without consent will generally need to be destroyed or deleted from the dataset. That puts the screening pass on a scraped or bulk-sourced dataset ahead of training.
Data already held for another purpose carries the APP 6 test with it into the training set. The guidance states that in the context of training generative AI models, updating a privacy policy or providing notice, by themselves, will generally not be sufficient to change reasonable expectations. That applies to the use of personal information that was previously collected for a different purpose.
Accountability sits with the developer at any scale. A developer subject to the Privacy Act must take reasonable steps to implement practices, procedures and systems that ensure compliance with the Australian Privacy Principles and any binding registered APP code. It must also be able to deal with related inquiries and complaints.
What notice do you give once an AI tool is in use
Adopting a commercial AI product carries a notice obligation. The OAIC states that organisations should update their privacy policies and notifications with clear and open information about their use of AI. They should also make sure that any public-facing AI tool, such as a chatbot, is clearly identified as such to users. Where that chatbot collects what people type into it, the collection has to comply with APP 3 and APP 5. The guidance states that individuals should be made aware they are interacting with an AI system and not a human.
What the notice says has to match what the product does with the information, so it gets written from the answers to the purpose test and the due diligence. A privacy policy drafted before the AI rollout does not describe what the business now does with customer data. A support chatbot that is never identified as automated does not meet what the OAIC expects on transparency and fair collection, whatever the use and disclosure analysis concludes.
What a privacy impact assessment for one AI tool covers
A privacy impact assessment run against one AI product and the two OAIC guidance documents answers five questions in order.
- What personal information the product will see, and what it will generate or infer.
- Whether that handling sits inside the purpose the information was collected for, or needs consent or an established reasonable expectation under APP 6.
- What the product's terms and settings give the developer access to, checked against the OAIC's checklist for selecting an AI product.
- Who inside the business can see the inputs and the outputs, and which staff are approved to enter personal information at all.
- What the privacy policy and any user-facing notice have to say once the first four answers are settled.
Where the work starts
A tool nobody recorded answers none of the five questions, so the inventory of what staff are already using comes before the assessment. The AI security assessment runs that first pass. It covers shadow AI discovery through tenant, network and app-consent review, and an AI usage policy setting out what is approved for use with business data. It also defines access boundaries for each AI tool in scope, so a chatbot or a plugin cannot reach data it should not. The purpose test and the privacy policy wording then run against the tools the business actually has.