A fairer start
A healthy life
A sustainable future
Our innovation methods
Mission Discovery
Policy
The Mission Studio
Nesta in a nutshell
Our strategy
Our people
Our history
Blogs
Reports
Events
Project updates
Twitter
Facebook
Linkedin
Bluesky
Nesta is a research and innovation foundation. We apply our deep expertise in applied methods to design, test and scale solutions to some of the biggest challenges of our time, working across the innovation lifecycle.
Creating an independent assurance process for auditing and assuring the safety of humanitarian AI tools
Principal Researcher, Centre for Collective Intelligence Design
Aleks Berditchevskaia is the Principal Researcher at Nesta’s Centre for Collective Intelligence Design.
Product design lead, Centre for Collective Intelligence Design
Esther is the product design lead for the Centre for Collective Intelligence Design at Nesta, harnessing the power of collective intelligence to design digital products & services.
Delivery and Product Manager, Centre for Collective Intelligence Design
Alex was a delivery and product manager for the Centre for Collective Intelligence Design.
Director of the Centre for Collective Intelligence Design
The Centre for Collective Intelligence Design explores how human and machine intelligence can be combined to develop innovative solutions to social challenges
Twitter
Facebook
Linkedin
Bluesky
Artificial intelligence (AI) holds massive potential for the humanitarian sector – from computer vision models that automate damage assessment in the immediate aftermath of a crisis, to chatbots that help vulnerable communities and frontline workers.
Despite this, the sector is struggling with limited technical expertise and low AI literacy. Findings from our research with Sphere indicate that 57% of humanitarians either aren’t aware of or don’t have any guidance about responsible AI in their organisations. This opens up humanitarian organisations to significant risk if things go wrong.
Since August 2024, Nesta’s Centre for Collective Intelligence Design has been exploring the value of an AI Safety Label – an independent assurance process for auditing and assuring humanitarian AI tools. Our process involved three key elements:
AI Safety Label animation
Our aim was to design an approach that would boost confidence for non-technical users whilst ensuring organisations were taking a responsible approach to AI deployment.
What makes our approach unique in the landscape of AI assurance is that it involves communities in assessing the social acceptability of risks of AI tools. We see the AI Safety Label as a community trustmark that can help organisations navigate risks while also fulfilling their accountability commitments to crisis-affected communities.
In November 2024, we set out to test the feasibility of our community trustmark approach: could we have meaningful conversations with crisis-affected communities about AI tools that could impact their lives? And would their recommendations provide useful information for humanitarian organisations as they decide whether or not to use a given AI tool?
We travelled to Antakya in southern Türkiye to gather community input on the use of Automated Damage Assessment (ADA), an AI tool developed by the Red Cross to identify damaged buildings using satellite images following natural disasters. Antakya was devastated by a series of earthquakes in early 2023 – it was reported that more than 80% of all buildings collapsed.
Working with a local expert facilitator, we held two deliberative workshops with a diverse group of 24 residents. Our approach draws on a collective intelligence method known as deliberative polling. It combines easy-to-understand, interactive animations, facilitated group discussion, expert Q&A and polling. The sessions are delivered through our bespoke collective intelligence platform Zeitgeist.
We know that people are more able to meaningfully engage with complex questions about technology when they’re asked about specific use cases, rather than abstract concepts. Building on this idea, we created bite-size animations about:
In between demonstrations of the bite-size content, participants discussed how they felt about humanitarians using a tool like ADA in their local context, drawing on their recent experiences of the earthquake. They weighed up the benefits of the tool (eg, faster, safer damage assessment) and the risks (eg, underestimation of damage leading to unfair allocation of resources). Throughout these discussions, we captured their views through interactive polls and recordings.
Too often, we’ve heard that it’s too difficult to involve communities in complex decisions about technology or that when it’s done, it’s hard to do it well. Our previous work on localising AI for crisis response and the results from this project show this isn’t true. It is possible to engage frontline communities in meaningful conversations about AI. In contrast to expectations that “communities will say no”, these processes can open up a mandate to experiment with AI as long as they have the appropriate guardrails in place. These were our key takeaways from the deliberative workshops:
Despite the risks, 92% of participants were positive about the use of the tool to improve future response operations in their local context.
The top two mitigations they wanted organisations to implement first were staff training and ongoing impact monitoring.
Workshop participants ranked safety measures based on which ones they thought humanitarian organisations should prioritise
A digital survey asks participants to rank AI safety measures. Responses prioritize training staff on the AI tool, monitoring its impacts, and verifying results before decision-making. A sidebar summarizes the top-ranked measures, including AI strategy development and incident reporting. The image emphasizes safety in AI implementation.
Conversations about responsible AI often end up assessing tools in comparison to an idealised, perfect version of the technology. In contrast, our experience shows that people are pragmatic – they compare the technology to the status quo and can be discerning about the limitations of human processes that might be enhanced by AI. In Antakya, workshop participants spoke about the impacts of the earthquake on frontline responders, who were often local people who had themselves been affected.
“In large-scale disasters like the Kahramanmaraş earthquake, humans are deeply affected, which could make their assessments less accurate."
Most importantly, workshop participants told us they enjoyed taking part (96%) and thought it was important that humanitarians involved people like them in decisions about using AI (92%).
“I am educated to primary school level, and as this kind of ordinary person I've never had the chance to learn what AI is. And now after this workshop I understand it, and I know how to explain it to others."
We used the results from the workshop to create a prototype AI Safety Label for the ADA tool to support damage assessment in southern Türkiye. The Label summarises the community input on the acceptability of different risks and the safety measures they want to see implemented, as well as a final recommendation about whether the tool should be used. It also documents the results of a technical evaluation of the tool carried out by the Red Cross.
A laptop screen shows an AI Safety Label for an Automated Damage Assessment (ADA) tool used for building damage assessment and earthquake response in Hatay, Turkey. The interface highlights key review steps: AI tool introduction, technical review, and community feedback.
A detailed breakdown of community responses to an AI damage assessment tool for NGOs and emergency responders. A bar chart visualises satisfaction levels, while key concerns include explainability, speed, and usability. Transparency and better guidance are identified as areas for improvement.
A webpage titled "Technical Review" for an AI tool named Automated Damage Assessment (ADA). It explains that ADA uses a convolutional neural network to assess disaster impact via satellite imagery. The tool is trained on a dataset of 859,000 buildings and follows a four-tier damage classification system. Icons for "Computer Vision" and "Deep Learning" highlight key technologies. A section on strengths and benefits emphasizss its robust dataset for effective damage assessment.
These are the first steps in our efforts to develop a robust and replicable assurance process that foregrounds community voices in decisions about how AI tools are used in high-risk settings. We believe that an independently-verified community trustmark is a critical gap in current conversations about AI assurance.
All the humanitarians we spoke to agreed that practical guidance was desperately needed. They also agreed that community accountability was an important part of getting AI implementation right in the sector. But they also couldn’t see their organisations signing up to a community trustmark unless there was strong endorsement by donors or demonstrator pilots, with big humanitarian organisations leading the way.
We believe the sector needs to move in this direction as part of ongoing conversations about standards and principles for responsible AI. Over the next year, we’ll continue developing this concept, building on our work in the humanitarian sector and expanding to other contexts, such as the UK public sector. In the meantime, look out for upcoming events about standards and responsible AI for humanitarians from our partners Sphere and CDAC Network.
We’re grateful to our advisors Kate Conroy, Kasia Chmielinski and Mallory Durran for their guidance throughout this work and our partners at Sphere, Data Friendly Space and CDAC Network. This project was made possible with the support of the UK Humanitarian Innovation Hub.
Principal Researcher, Centre for Collective Intelligence Design
Aleks Berditchevskaia is the Principal Researcher at Nesta’s Centre for Collective Intelligence Design.
Product design lead, Centre for Collective Intelligence Design
Esther is the product design lead for the Centre for Collective Intelligence Design at Nesta, harnessing the power of collective intelligence to design digital products & services.
Delivery and Product Manager, Centre for Collective Intelligence Design
Alex was a delivery and product manager for the Centre for Collective Intelligence Design.
Director of the Centre for Collective Intelligence Design
The Centre for Collective Intelligence Design explores how human and machine intelligence can be combined to develop innovative solutions to social challenges
We extend our impact through two specialised units that help people and organisations to solve complex problems and achieve their goals.
BIT helps clients from government, nonprofits and the private sector to improve people’s lives through our empirical problem solving and deep understanding of human behaviour.
Challenge Works designs and runs challenge prizes to spark innovation in science, technology and society.
Join our mailing list to receive the Nesta Edit, our regular newsletter showcasing how we design, test and scale solutions to some of society’s biggest challenges, with updates from Nesta, BIT, Challenge Works and the wider innovation sector.
* denotes a required field
You can unsubscribe by clicking the link in our emails where indicated, or emailing [email protected]. Or you can update your contact preferences. We promise to keep your details safe and secure. We won’t share your details outside of Nesta without your permission. Find out more about how we use personal information in our Privacy Policy.
Leave a Reply