Governing AI agents: what changes when a system acts, not just advises

Governing agents is an advanced governance problem. It tends to arrive after an organisation has the straightforward work in hand, a register of its systems, risk assessments, derived controls and policies, and is extending its practice into the harder cases. It sits alongside impact assessments and incident handling as one of the more demanding forms of governance a maturing function takes on, not as the whole of the work.

What actually changes

The shift from advice to action changes three things about how you oversee a system.

The unit of oversight moves from the output to the action. With an advisory system you review what it produced. With an agent you have to bound what it is allowed to do before it does it, decide which actions it may take without a person and set which actions always stop for approval. Reviewing a good answer after the fact is not oversight if the agent has already sent the email or changed the record.

The surface area grows with every tool the agent can reach. An advisory model is largely contained by its prompt and its output. An agent is defined by its permissions, the systems it can read from and write to, the credentials it holds and the other agents or services it can call. OWASP’s work on agentic threats describes categories that only appear once a system acts, among them the misuse of the tools an agent is given and the compromise of the memory or context it relies on to decide what to do next. The risk lives in what the agent can touch, not only in what it says.

Accountability does not move. This is the part that senior owners ask about first and it has a settled answer. Australia’s Voluntary AI Safety Standard puts it plainly in its first guardrail: leaders cannot delegate or outsource accountability for the safe and responsible use of AI. An agent acting on its own does not become the accountable party. A named person remains answerable for what the agent does, and the governance exists to keep that person in genuine control of a system that is designed to act without them at each step.

The questions being asked at senior level

These are the questions that reach a board or an audit committee once agents are in use, and the guidance that speaks to each.

Who is accountable for this agent, and can they actually control it? This is Guardrail 1 of the Voluntary AI Safety Standard, accountability and governance, read together with Guardrail 5, meaningful human oversight and the ability to intervene. Oversight of an agent means a person can see what it is doing and can stop or override it, not that a person signed off once at the start.

What is this agent allowed to do, and what has it done? This is a records question before it is a technical one. Guardrail 9 of the standard asks you to keep records that let a third party assess your work against the guardrails. For an agent that means recording its purpose, its permissions, the actions it may take on its own, the actions that stop for approval and the human owner, and keeping that record current as the agent changes.

Have we assessed the risk of letting it act? The NIST AI Risk Management Framework organises this as Govern, Map, Measure and Manage, and its Generative AI Profile speaks to the risks that generative and agentic systems introduce. For a high-risk agent the assessment is the impact assessment you would run for any consequential system, extended to cover the agent’s autonomy, its reach into other systems and the way it fails.

Where the guidance is moving

The current Australian anchor is the Voluntary AI Safety Standard and its ten guardrails, aimed at organisations deploying and using AI. The Australian Government has also consulted publicly on mandatory guardrails for AI in high-risk settings. Internationally, the NIST AI Risk Management Framework and its Generative AI Profile, ISO/IEC 42001 as a management-system standard and OWASP’s agent-specific threat work give the allied picture. None of these is agent-only, and that is the point. Governing agents is the existing governance you already run, applied to systems that act, with the added obligations that action brings.

Where Aicura fits

Aicura is your AI Register, and it holds your agents as their own kind of entry in the same register as your systems. It surfaces each agent’s purpose, permissions, the actions it may take on its own and its human owner. It versions that record as the agent changes and prompts you when a review is due. Alongside it, Impact Assessments carry the risk work for agents that warrant it, and Incidents give you the place to record when an agent does something it should not have.

Aicura supports the work and surfaces the picture. It does not certify your agents, sign off their use or decide whether an agent is safe to run. The accountable owner makes that call, and Aicura gives them the record and the evidence to make it and to show it to the board, the auditor or the regulator later.

Common questions

Is an AI agent just a chatbot? No. A chatbot returns answers for a person to read and act on. An agent takes actions towards a goal, calling tools and changing other systems with less of a person between each step. The governance question is the same in kind but larger, because the agent acts rather than advises.

Do we need a separate framework to govern agents? Not a separate framework. The Voluntary AI Safety Standard, the NIST AI Risk Management Framework and ISO/IEC 42001 already apply. Governing an agent is applying that existing work to a system that acts, and adding the records, the oversight controls and the risk assessment that autonomy calls for.

Who is accountable when an agent acts on its own? A named person in the organisation. The Voluntary AI Safety Standard is direct that accountability for AI cannot be delegated or outsourced, and that does not change because a system is designed to act without a human at each step.

Where do we start? Record your agents first, so you know what exists and what each one can do. From there, assess the higher-risk agents and set the oversight controls that let a person intervene. The companion guides on assessing and overseeing an agent and what to record about an agent walk through each.

Sources

• Voluntary AI Safety Standard, Department of Industry, Science and Resources, industry.gov.au/publications/voluntary-ai-safety-standard
• AI Risk Management Framework (AI RMF 1.0) and Generative AI Profile, NIST, nist.gov/itl/ai-risk-management-framework
• Agentic AI: Threats and Mitigations (v1.0), OWASP GenAI Security Project, genai.owasp.org
• ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system

A note on this page

This guide explains current Australian and allied guidance on governing AI agents and how to approach the work. It is general information, not legal advice, and it does not tell you whether a particular agent meets a particular obligation. For how the guidance applies to your organisation and your agents, read the primary sources above and take your own professional advice.