Safeguarded AI

About the programme

The UK's Advanced Research + Invention Agency (ARIA) recently published v2.0 of its Safeguarded AI programme thesis. It's a substantial evolution of the original 2024 vision:

In light of how rapidly frontier AI has advanced since early 2024, the £18m originally earmarked for fine-tuning frontier models for verification-friendly behaviour — originally (but no longer) thought to be a pre-requisite to employing the mathematical toolkit developed elsewhere in the programme — has now been reallocated. Furthermore, the Safeguarded AI focus has shifted towards accelerating societal resilience — expanding the speed, scale, and scope of AI-enabled mathematical modelling and formal verification.

HASH's role

Real-world applications

Building on our historical agent-based simulation modeling, process modeling, and graph AI work, HASH has been leading a series of projects applying Safeguarded AI methods to safety-critical cyberphysical domains, including chemical and biopharmaceutical supply chains and manufacturing — a follow-up to our earlier work with the University of Oxford. As part of this we've been working with firms in industry to directly develop, test and deploy Safeguarded AI-ready solutions in the real-world.

Many of our existing open-source technologies, such as hgres — our typed, multi-temporal, provenance-aware graph data engine — fit neatly alongside other parts of the Safeguarded AI toolkit, and serve as a strong basis for these pilots.

Technical area lead

We've also taken on the role of leading TA1.3, the Interaction Paradigms technical area for the entire Safeguarded AI programme, building on our work at HASH helping domain experts encode their understanding of complex environments at scale (in the form of HASH simulation models), and extending this to formal verification and specification of all kinds of systems. We're also looking at how humans and AI agents collaborate on authoring, eliciting, and auditing the formal artefacts (world-models, specifications, certificates) that the rest of the Safeguarded AI toolkit operates over (for example, via Brunch).

How this aligns

HASH helps organizations build realistic representations of their domains, quickly bootstrapping hyper-accurate, incredibly detailed knowledge graphs and causal process graphs in an AI-assisted way. These world models have many uses, including for traditional simulation, decision support, process optimization and automation... but in the Safeguarded AI context we're excited about extending them in a way that allows HASH as a platform to provide probabilistic proofs of safety, unlocking Large Language Models for safety-critical domains where they cannot otherwise reliably be utilized with confidence.

The desiderata in Safeguarded AI's technical toolkit — machine-native representations of programs/specs/proofs, structured and versioned collaboration over those artefacts, proof-aware writes, fast parallel checking, local-first execution — closely track a stack we've been building for years and have begun releasing publicly over the past month:

• →

  hgres — a backend-agnostic, strongly-typed graph data engine that decouples query and schema semantics from the underlying database, with first-class multi-temporal support (announcement).

• →

  HashQL — a multi-temporal hypergraph query language with a compiler-first design that places computation transparently across heterogeneous execution backends (preview).

• →

  SemType — a specification and open registry of semantic types, providing a shared vocabulary that humans and AI agents can use to exchange structured data unambiguously (announcement).

• →

  The HASH platform itself, which composes these into typed, provenance-aware knowledge and process graphs for organizations, people, and the AI agents acting on their behalf.

Our parallel work on Petrinaut — a browser-based editor and simulator for Petri nets, a mathematical formalism for structuring, auditing and verifying concurrent, distributed systems — also feeds into the Safeguarded AI programme's mathematical assurance toolkit, as well as underpinning elements of our research into process foundation models.

While the Safeguarded AI v2 thesis has stepped back from the original "world-modelling ML" sub-area; we believe the question still matters and are continuing to invest in it (see our recent posts on Graph-Based World Models, and Towards Process Foundation Models).

Get involved

We're now looking for industry partners beyond chemicals and biopharma, particularly across supply chains, manufacturing, and adjacent cyber-physical verticals, who are interested in testing out the Safeguarded AI technology.

If you think AI-enabled formal methods could move the needle on your operations, please get in touch via our website to learn more about participating.

We're also recruiting AI/ML researchers with backgrounds in graph machine learning, reinforcement learning, and formal verification (amongst others). If this is you, or you have complementary expertise you'd be excited to share, check out our open roles to learn more.