Ask a minister why a government AI project failed and you will usually hear about the technology: bad data, weak engineers, a vendor that oversold. You may have said something similar yourself. Now look at the public-sector technology disasters of the past decade, the ones that brought down a government, produced a Royal Commission and forced a parliament to quash criminal convictions by statute. Almost none of them involved sophisticated technology. The code was ordinary. What failed was everything around it.
That is the uncomfortable starting point for anyone buying AI for the state in 2026. The question is rarely whether the model works. It is whether the institution deploying it can be trusted to notice when it does not.
The code was ordinary
Take Australia’s Robodebt. The scheme took a person’s annual income as reported to the tax office and spread it evenly across fortnights, then compared the result with what they had declared to Centrelink. Anyone whose earnings varied from one fortnight to the next, casual workers and students among them, could look as if they had been overpaid. Debt interventions rose from about 20,000 in 2015-16 to nearly 800,000 in 2016-17, a fortyfold jump in a single year. The Federal Court found the scheme unlawful in late 2019, and on 7 July 2023 the Royal Commission concluded that Robodebt was "a crude and cruel mechanism, neither fair nor legal."
Crude is the important word. This was arithmetic, not artificial intelligence. A spreadsheet could have done it.
So why did it run for years? Not because the maths was hard to check. It ran because the people receiving the letters had far less power than the people sending them, and because nobody inside the system was rewarded for asking whether the debts were real.
Who the system answered to
In the Netherlands, the tax authority’s childcare benefits system treated small errors, a missing signature or a mistake of around €100, as grounds to demand back a full year of subsidy, sums that could reach €20,000. Families were evicted, marriages ended, people lost their jobs. Amnesty International later found that "Dutch citizenship: yes/no" was a category in the risk classification model, and that non-Dutch nationals received higher risk scores. On 15 January 2021, Mark Rutte’s entire cabinet resigned. Affected parents were offered at least €30,000 each in compensation.
A year earlier, in February 2020, the District Court of The Hague had already struck down SyRI, a separate system that combined government databases to give citizens a fraud risk score. The court held that the legislation behind it breached Article 8 of the European Convention on Human Rights, because the use of SyRI was insufficiently transparent and verifiable. People did not know their scores, so they could not challenge them, and the system was reportedly used only in low-income neighbourhoods and in municipalities with many non-Western immigrants.
Notice what the judges objected to. Not the accuracy of the model, but its secrecy, and the absence of any way for a citizen to answer back.
When an institution trusts the machine over its people
The UK Post Office scandal is the purest case, because Horizon was not AI at all. It was an accounting system supplied by Fujitsu. Between 1999 and 2015, around 1,000 sub-postmasters were prosecuted and convicted on the strength of its data, more than one a week for sixteen years. In December 2019, Mr Justice Fraser found that bugs, errors and defects could cause apparent shortfalls in branch accounts, and likened the Post Office’s insistence that Horizon was robust to maintaining that the earth is flat. In May 2024, Parliament quashed hundreds of convictions by statute. The public inquiry’s first report, published in July 2025, said at least 13 suicides could be linked to the scandal.
Why did it take twenty years? Partly because, in England and Wales, software-generated evidence is treated as accurate unless someone shows otherwise. The Ministry of Justice only opened a call for evidence on that presumption in January 2025.
If the law assumes the machine is right, who exactly is going to prove it wrong? A sub-postmaster behind the counter of a village shop?
What regulation can and cannot do
Europe’s answer is the AI Act, which entered into force on 1 August 2024. It classifies as high-risk any AI used by public authorities to evaluate eligibility for essential public assistance benefits and services, or to grant, reduce, revoke or reclaim them. That description fits Robodebt and the Dutch benefits system almost word for word. Yet those obligations, originally due on 2 August 2026, were pushed back by the Digital Omnibus, in force since 27 July 2026, to 2 December 2027.
Don’t get us wrong. The Act matters, and a later start with clearer standards may produce better compliance than a rushed one. But if you are a government outside the European Union, in Colombo or Banjul or Chișinău, you cannot outsource your judgment to Brussels’ timetable. And even inside the EU, a regulation does not decide what goes into your contract, who reviews a flagged case, or whether a citizen can see why a payment stopped.
That gap, between what a vendor promises and what an institution can actually govern, is where trust is won or lost.
What earns trust
Look at the systems that have lasted. Estonia’s X-Road, the data exchange layer behind its e-government, deliberately has no central database. Each ministry controls access to its own data, and the once-only principle means a citizen hands over a piece of information once rather than to every agency in turn. It now carries more than 3,800 e-services and about 2.2 billion transactions a year. Its architects made a governance decision first and a technical one second.
India’s Aadhaar offers a different lesson. In September 2018 the Supreme Court upheld the identity system for delivering welfare, but struck down the provision that allowed companies to use it simply by signing a contract. A limit imposed from outside made the system more defensible, not less. The Netherlands, chastened, launched a public Algorithm Register in December 2022, where government bodies describe the algorithms they use and how they work. Participation is still voluntary. That tells you how hard this is, even for a country that has already paid the price.
None of this is glamorous. Trust is earned by systems a citizen can see, question and appeal, run by officials who are rewarded for finding errors rather than defending them, under contracts that keep the state, not the vendor, in charge of the data and the decision. It is earned by being honest at the start about what the technology can do, what it cannot, and who carries the cost when it is wrong.
That is why we think the most important work in public-sector AI happens before a single model is switched on: in the business case, the procurement, the oversight design and the appeal route. It is also why our Tech Advisory practice sits on the government’s side of the table and has no software to sell.
The model is the easy part. The institution is the product.
