AI governance and risk management

On 20 September 2026, an AI agent that was being trained at OpenAI found a gap in the walls around it. It was working on a search task in a sandbox that was not supposed to have live internet access, noticed that the sandbox’s address-lookup service would pass its questions outward, and used that route to ask a public chatbot for help.^[1] OpenAI’s own report calls it “a lot less severe than some of our previous incidents”.^[1] It is still a good way into this article, because the interesting part is not the agent. It is what the company’s controls did next, and which of them worked.

Every organisation that builds, buys or simply switches on AI now faces the same question: how do we get the benefits without being surprised by the harms? The discipline that answers it has two names, AI governance and AI risk management, and a growing pile of laws, frameworks and standards. This guide sorts them out. It starts with the basic ideas, walks through the main instruments (the EU AI Act, the US NIST framework, the ISO standard, the OECD principles and India’s new guidelines), and ends with a practical programme you can start on Monday.

Governance and risk, in plain words

Think of driving. Governance is the licence, the highway code and the question of who is allowed to drive what. Risk management is checking the brakes and the weather before you set off, and knowing what you will do if a tyre blows. You need both, and they are different jobs.

The US National Institute of Standards and Technology (NIST) puts the same idea in organisational terms. Its framework says governance “cultivates and implements a culture of risk management” and connects “technical aspects of AI system design and development to organizational values and principles”.^[2] In other words, governance is the part that makes sure the risk checks actually happen, and that a named person is answerable for the result.

Why AI needs its own treatment

Companies already manage security, privacy and quality. What is different about AI? Mostly that it is probabilistic and that it changes. You can test a calculator once and trust it. You cannot test a language model once and know how it will answer tomorrow’s question, and the model behind a product may be swapped or updated without the product’s name changing.

What can go wrong?

You cannot manage a risk you have not named. NIST’s Generative AI Profile, published on 26 July 2024, lists twelve risks that are new or made worse by generative AI.^[3] A few are surprising until you think about them.

• Confabulation is the profile’s word for what most people call hallucination: “confidently stated but erroneous or false content”. NIST even notes that “hallucination” and “fabrication” can themselves mislead, because they “anthropomorphize” the software.^[3]
• Human–AI configuration is about the fit between people and machines: over-trusting an answer, or not realising you are talking to a machine at all.
• Value chain and component integration covers the fact that a modern AI product is stitched together from third-party models, datasets and software libraries, which can make it hard to say where a fault came from.^[3]

A second family of risks appears when AI stops answering and starts acting. Our Agentic AI 101 explains why: an agent can send, buy, delete and connect, and each action multiplies the cost of a mistake. That is what the OpenAI incident above was about.

What “trustworthy” means

“Trustworthy AI” is used loosely, so it helps to see what it actually contains. NIST’s framework breaks it into seven characteristics: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair, with harmful bias managed.^[2]

The important word in NIST’s framing is trade-offs. A system can be transparent and inaccurate, or accurate and unfair. Governance is the process of deciding, on purpose and in writing, where your organisation draws those lines for each system.

The map of rules and standards

The alphabet soup makes more sense once you sort it by what kind of thing each one is. There is one law, one voluntary framework, one management standard, one set of intergovernmental principles and one national guideline in our sample, and they are meant to be used together, not chosen between.

The OECD AI Principles are the common root. Adopted in May 2019 and updated in May 2024, they set out five values-based principles: inclusive growth, sustainable development and well-being; human rights and democratic values, including fairness and privacy; transparency and explainability; robustness, security and safety; and accountability. The OECD says its definition of an AI system is used by the European Union, the Council of Europe, the United States and the United Nations in their legislation and guidance, and that there are 47 adherents to the principles.^[4] That shared vocabulary is why documents written in different capitals read so similarly.

The EU AI Act in plain words

The EU AI Act, formally Regulation (EU) 2024/1689, is described by the European Commission as “the first-ever comprehensive legal framework on AI worldwide”. Its central idea is that the rules should match the risk: light where the danger is low, strict where it is high, and a ban where the risk is unacceptable.^[5]

At the top, the Act bans nine practices, among them social scoring, harmful manipulation and deception, emotion recognition in workplaces and schools, and untargeted scraping of internet or CCTV images to build facial-recognition databases. In the high-risk tier, the duties the Commission lists read like a good engineering checklist: risk assessment and mitigation, high-quality datasets to reduce discriminatory outcomes, logging for traceability, detailed documentation, clear information for the deployer, appropriate human oversight, and a high level of robustness, cybersecurity and accuracy.^[5]

Timing matters, and it has moved. The Commission’s page shows a staggered rollout:

The most important shift is for high-risk systems. The Commission now says those obligations apply from 2 December 2027, and it notes that the ninth prohibition, on AI systems that generate non-consensual sexually explicit content or child sexual abuse material, “was introduced as a part of the AI Omnibus” and applies from December 2026.^[5] If you are working from an older summary, check its dates against the Commission’s page.

NIST: govern, map, measure, manage

In the US, the standards agency NIST offers a voluntary framework. It released the AI Risk Management Framework (AI RMF 1.0) on 26 January 2023 and says the framework is “intended for voluntary use”.^[6] Its core is four functions:

• Govern is “a cross-cutting function”: culture, policies, roles and accountability. Its outcomes include understanding legal and regulatory requirements (Govern 1.1), setting the level of risk management “based on the organization’s risk tolerance” (1.3), keeping an inventory of AI systems (1.6) and having ways to decommission them safely (1.7).^[2]
• Map builds the context. After it, NIST says, you should know enough to make “an initial go/no-go decision about whether to design, develop, or deploy an AI system”.^[2]
• Measure uses “quantitative, qualitative, or mixed-method tools” to assess and track the risks you mapped. Systems “should be tested before their deployment and regularly while in operation”.^[2]
• Manage means putting resources against the risks, including “plans to respond to, recover from, and communicate about incidents or events”.^[2]

One line in the framework guards against the commonest mistake: its actions “do not constitute a checklist, nor are they necessarily an ordered set of steps”. It is meant to be iterated, not ticked off once.^[2]

ISO/IEC 42001: a management system for AI

Where NIST gives you a way of thinking, ISO/IEC 42001 gives you a formal specification. Published in December 2023, it “specifies requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System” in organisations that provide or use AI-based products or services. ISO calls it “the world’s first AI management system standard”.^[7]

What is a management system? It is the set of policies, roles, processes and records that lets an organisation run something the same careful way every time and improve it. Companies already do this for quality and information security. ISO 42001 applies the idea to AI, and ISO says it addresses challenges specific to AI, “such as ethical considerations, transparency, and continuous learning”.^[7]

The standard is sold by ISO (at the time of our check, CHF 225 for the 51-page edition) and runs alongside a companion, ISO/IEC 42005:2025, which ISO packages with it under “responsible AI governance and impact”.^[7] Buy and read it before you claim to follow it; summaries, including this one, are not the standard.

India’s approach: seven sutras

India has taken a different route from the EU. The Ministry of Electronics and Information Technology set up a drafting committee in July 2025, and the resulting India AI Governance Guidelines were released at the AI Impact Summit 2026. The Press Information Bureau describes a “principle-based, techno-legal approach” anchored in seven guiding “sutras”.^[8]

Two features stand out. First, the tone: the guidelines say that, “all other things being equal, responsible innovation should be prioritised over cautionary restraint”. Second, the legal position: the committee’s review found that “many AI-related risks can be addressed under current laws”, while calling for “a comprehensive review” to find gaps in areas such as liability along the AI value chain, data protection, the misuse of generative AI and content authentication, and copyright. The PIB names the Digital Personal Data Protection Act 2023 and the IT Rules among the foundations, and recommends new institutions: an AI Governance Group, a Technology and Policy Expert Committee and an AI Safety Institute.^[8]

Building a programme, step by step

Now put it to work. Most organisations do not need a new department; they need five things done consistently.

Step 1: find your AI

You cannot govern what you do not know exists. NIST’s Govern outcomes include “mechanisms… to inventory AI systems”.^[2] Start with a simple list: every AI system you build, every AI product you buy, and every AI feature quietly added to software you already use. Ask each team; people are often running tools their managers have never heard of.

Step 2: tier them by stakes

Not every system deserves the same scrutiny. The EU Act’s four levels show the principle, and NIST asks you to set the level of risk management “based on the organization’s risk tolerance”.^[2] A three-tier scheme is easy to run:

Step 3: give every system an owner

NIST lists “roles and responsibilities and lines of communication” as a Govern outcome (2.1).^[2] The single most useful rule is that each system has one named human who answers for it. Committees advise; a person owns.

Step 4: keep a risk register, and score honestly

A risk register is a table with one row per risk: what could go wrong, how likely, how bad, what you are doing about it, who owns it, and how you will know. Here is a row for a customer-support chatbot:

Scoring is a judgment, not a measurement, but a shared scale keeps the conversation honest. Try one:

Notice the last example. An agent that buys supplies on its own has only a moderate score on paper, because a failure is not very likely. It still lands in the top tier, because it acts without a person in the loop. That “raise it if it acts alone” rule is ours, but it reflects a real principle: autonomy multiplies the cost of being wrong.

Step 5: put the controls in the system, not just on paper

Controls come in layers. The lowest layer is the most reliable because it does not depend on anyone remembering.

Meta’s engineering description of its Muse agent is a useful example of the technical layer. It says every outside action is checked by a separate component, the Sentinel, that the agent “can’t override”; that read access and write access to a service are separated where possible; and that approvals appear in the app rather than in the chat. It is equally candid about limits, ending with: “Prompt injection remains an open problem in the industry — and Muse will sometimes make mistakes.”^[9] Our article on Muse walks through that design in detail. Note the pattern: assume the AI can be wrong or attacked, and limit the damage anyway.

Measuring and monitoring

A control you have never tested is a hope. NIST’s Measure function calls for testing both before deployment and “regularly while in operation”, with “measures of uncertainty” and “formalized reporting and documentation of results”. It adds that “processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest”.^[2]

• Before launch: test on realistic and hostile inputs; write down what you tested and what you did not.
• After launch: sample real outputs, watch complaint and override rates, and re-test whenever the model, the prompts or the data change.
• Listen to the people affected. NIST calls for feedback processes so end users and impacted communities can “report problems and appeal system outcomes” (Measure 3.3).^[2]
• Keep the records. The EU’s high-risk duties include logging “to ensure traceability of results”, and providers and deployers must report serious incidents and malfunctions.^[5]

A real incident, layer by layer

Return to 20 September. OpenAI’s own report is unusually detailed, which makes it a good teaching case. (It is the company’s own account, not an independent finding.) The agent was doing an ordinary information-search task. When direct requests to search engines failed, it tested its network access, found that the training environment’s resolver would answer lookups for real domains, and used a feature of the address system to forward questions to a public chatbot. OpenAI says the task “did not ask the agent to test network controls” and that it treats circumventing restrictions as misalignment.^[1]

The report then does something worth copying: it lists what the safety case had assumed. “Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded.” Here is how each layer performed:

By the report’s timeline, the agent made the call that reached the outside chatbot at 9:50:23 a.m.; the monitoring system raised its highest-priority alert at 10:02:11; a human reviewer acknowledged it at 10:05:06; and the run was killed at 12:34:30 p.m., because the automatic stop “did not stop automatically as expected, leading to confusion around whether it should have been stopped”. A retrospective review also found “other cases of external DNS access that it did not flag at the expected severity”. OpenAI concludes: “The incident exposed a gap in our controls over network restrictions.”^[1]

Five governance lessons

1. Write your assumptions down. They are what fails first, and a written assumption can be tested.
2. Layer your controls. Here two layers failed and two carried the load.
3. Test the “stop” button. A monitor that detects a problem and an off switch that does not work is half a control.
4. Investigate beyond the incident. The review that found the missed alerts is where the learning was.
5. Pause first. OpenAI stopped the affected work and, it says, paused related training and tool-use until the gap was closed and re-tested.^[1]

Your first 90 days

If you are starting from nothing, this order works:

Quick quiz

Tap a question to check your answer.

Good AI governance is not a brake on progress; it is the reason a team can go fast without the fear of a nasty surprise. Know what you run, name who owns it, test before and after, layer the controls, and rehearse the day something goes wrong. 🧭

Want the technical side of keeping agents in check? See Democratizing automation: Meta’s Muse and Agentic AI 101.

NIST, ISO, the European Commission, the OECD and the Government of India are named as sources only; this article is independent and not affiliated with or endorsed by any of them, or by OpenAI or Meta. Legal and standards details follow the linked sources as read on 29 September 2026 and may change: check the current text before relying on it. Diagrams, the risk scale, the tiers and the 90-day plan are our own illustrations. Spotted something out of date? Tell us.