Realising the Potential of Sociotechnical Approaches to AI Evaluation

This publication is licensed under the terms of the Creative Commons Attribution License 4.0 which permits unrestricted use, provided the original authors and source are credited.

Introduction

This Expert Analysis is based on a discussion at a joint CETaS–AISI virtual workshop hosted on 7 November 2024. The workshop focused on how to realise the potential of sociotechnical approaches to AI evaluation, involving 38 UK and international experts from across government, academia, industry and civil society. The discussion topics were inspired by “Evaluating Malicious Generative AI Capabilities: Understanding inflection points in risk,” CETaS Briefing Papers (July 2024). We would like to thank the workshop participants for their invaluable insights, and particularly acknowledge the contribution of our keynote presenter, Laura Weidinger.

Over the past year, the discourse on AI safety has increasingly centred on evaluation as a way to mitigate the risks posed by advanced generative AI (GenAI) systems. Approaches to evaluation vary from red-teaming to automated solutions, to human-participant studies. As the field evolves, efforts to establish best practices have become a priority for experts. With different AI safety hubs emerging around the world, it is crucial that experts maintain a holistic understanding of the AI threat landscape, enabling them to effectively address key areas of concern.

As explored in the CETaS Briefing Paper, GenAI systems can uplift malicious actors’ capabilities in numerous ways. The paper described a series of inflection points in risk across three threat domains: malicious code generation, radicalisation, and weapon instruction and attack planning. Unlike previous work focused primarily on technical changes in AI capabilities, the paper took an intelligence-led approach by addressing: malicious actors’ preferences and readiness to adopt certain technologies; in-group characteristics that may inform their interactions with AI systems; and systemic factors that are crucial in shaping the broader operating context. 

Landmark discussions on AI safety took place in San Francisco in November 2024 and will continue at the AI Action Summit in Paris in February 2025. Accordingly, it is more important than ever that AI developers, AI evaluators and the national security community find common ground and adopt shared approaches to these challenges.

Sociotechnical approaches to AI evaluation

Participants in the workshop attempted to establish a shared definition of sociotechnical approaches to human-machine interaction and AI evaluation. According to research by Weidinger et al. (2023), in AI evaluation, sociotechnical approaches address risks linked to:^[1]

• Technical components and system behaviours (e.g. a propensity to reproduce harmful stereotypes in images or utterances).
• Interactions between technical systems and human users (e.g. challenges for data annotators who are exposed to harmful model outputs).
• Systemic and structural factors that influence model capability and human interactions (e.g. increasing homogeneity in knowledge production and creativity).

Weidinger et al. assert^[2] that the sociotechnical lens gives rise to questions such as:

• Who is interacting with an AI system, and in which institutions?
• Which workflows and processes stem from these interactions?
• What do people use an AI system for, and how does this differ from developers’ intentions?
• How does an AI system function for people who its creators did not envisage as primary users?
• What points of failure emerge when there is widespread adoption of an AI system?

As Clark Barrett and his co-authors argued in a 2023 paper on the risks of GenAI,^[3] “human activity is remarkably adaptable, nuanced, and context-driven, whereas computational algorithms often exhibit a ‘rigid and brittle’ nature.” Bridging the gap between technical solutions and the social requirements for GenAI deployment is central to designing an effective evaluation ecosystem.

AI evaluation methods

Weidinger et al. have stated^[4] that while there is a logic to working ‘in to out’ – i.e. starting with model capabilities and ending with their impact on institutions and society – there is also a need to do the opposite. The latter involves assessing the real-world failure modes of AI systems and investigating the key factors that drive their adoption in different contexts, to ground and validate capability evaluations that might depart from the status quo.

Splitting AI evaluation into goals and methods clarifies the role of sociotechnical approaches in a broader context (Weidinger et al.):^[5]

1. For successful ‘hill-climbing’ (making small adjustments iteratively, to optimise performance based on feedback from evaluation metrics), benchmarks provide a useful performance metric for AI capabilities.
2. To explore likely failure modes, AI red-teaming appropriates methods from the cybersecurity domain, by identifying a model’s weak points so they can be patched.
3. For the goal of understanding the inner workings of an AI model, mechanistic interpretability can be helpful – although the science of machine behaviour remains a live and developing area.
4. For the goal of providing assurance (i.e. improving assessments of whether a model is safe), sociotechnical approaches may show the most promise given how they account for context.

Weidinger et al. also point out that one nascent approach is automation and simulation:^[6] testing AI systems in environments that emulate real-world conditions or specific task settings. Despite the complexity of grappling with lived human experiences, simulation could offer a partial solution to the challenge posed by human reviewers’ prolonged exposure to harmful real-world content during evaluation.

Notwithstanding the progress being made in these areas, there remain numerous gaps in AI evaluation methods. For example, research into interaction harms is accelerating but could be outpaced by the increasing volume and variety of people engaging with AI systems. Multi-modal evaluation is another promising area, but it is at an early stage and subject to the uncertainty caused by a growth in the applications of autonomous AI agents.

There is no single solution that fits all the complex layers of AI evaluation. However, as per Weidinger et al.,^[7] 85.6% of evaluations focus on capability and only 5.3% and 9.1% on human interaction and systemic impact respectively. There is a need to build on the goals and methods associated with sociotechnical AI evaluation.

Inflection points in risk and malicious actor uplift

Given the increase in both the accessibility of GenAI systems and the potential impact of their use by malicious actors, experts require new conceptual frameworks to address the risks. In this regard, inflection points are valuable in understanding and forecasting the heightened risk in certain uses of GenAI systems. The concept of inflection points encapsulates how risk levels may quickly rise or otherwise change at certain moments, and reflects the reality that risks can increase in a non-linear manner, with sudden shifts resulting from various factors (e.g. technological breakthroughs, regulatory changes or new uses of AI). Just as it is important to work both ‘in to out’ and ‘out to in’, it is vital to balance forward-thinking risk identification with efforts to work backwards from undesirable sociotechnical outcomes.

Greater clarity about inflection points and risk thresholds would be useful in persuading stakeholders to reach a consensus on the points at which mitigation measures will be needed. This may align with intelligence-led frameworks in the national security context, which are often constructed around threat levels. For example, the UK categorises the terrorism threat level as either low, moderate, substantial, severe or critical.^[8]

One caveat to this discussion is that the dual-use nature of many AI capabilities means there will also be a positive aspect to certain inflection points. Therefore, developers may see an inflection point in risk as an equally important innovation opportunity. For instance, inflection points in malicious code generation may correspond with those in autonomous agents for cyber defence. Similarly, a considerable improvement in GenAI systems’ social awareness and persuasiveness may be a significant concern from a radicalisation perspective but a boon from a marketing perspective, as it would be advantageous for applications like political fundraising. Given that there is no intrinsic threat to an AI model’s persuasiveness, risk assessments should balance the identification of bad actors deploying GenAI for harmful purposes against benign commercial use cases.

Inflection points and indicators 

As discussed, the workshop focused on the threat areas of malicious code generation, radicalisation, and weapon instruction and attack planning, while also addressing systemic factors. The aim was to scrutinise the inflection points covered in the CETaS Briefing Paper and identify new ones. The tables below summarise the inflection points of greatest interest to participants in the workshop, along with potential indicators that these points have been reached.

Table 1. Malicious code generation 

Table 2. Radicalisation 

Table 3. Weapon instruction  

Table 4. Systemic factors linked to inflection points  

Conclusion

The workshop highlighted the need for greater scrutiny of how technical, organisational and societal factors interact and mutually reinforce one another to amplify AI risks. Capability-centric perspectives on AI adoption have merits, but one should couple them with intelligence-led threat assessments of malicious actors’ tradecraft and operations.

No single community can tackle this challenge on its own – the CETaS-AISI workshop is just one of many dialogues aiming to broaden the coalition of stakeholders in the endeavour. Such efforts should evolve alongside technical and sociocultural shifts, and proceed with an open mind as to which combination of evaluation approaches will have the best chance of mitigating AI harms.

The views expressed in this article are those of the authors, and do not necessarily represent the views of The Alan Turing Institute or any other organisation. 

References