Bias in Military Artificial Intelligence and Compliance with International Humanitarian Law

Acknowledgments

SIPRI and the authors express their sincere gratitude to the German Federal Foreign Office for its generous financial support for this project.

The authors also wish to thank SIPRI colleagues Alexander Blanchard, Vincent Boulanin, Netta Goussac and Dustin Lewis for their invaluable insights and feedback throughout the project.

In addition, the authors thank all the experts who generously shared their knowledge and perspectives in the workshop SIPRI organized in Stockholm on 3–4 February 2025: Max Benítez, Svenja Berrang, Abhimanyu George Jain, Martin Hagström, Georgia Hinds, Arthur Holland Michel, Arnav Jagasia, Kwang Ian Leong, Hayley Ramsay-Jones, Samuel Segun, Sarah Shoker and Nicolas Wimberger. The authors are also grateful for insights and feedback provided at various stages by Alexander Breitegger, Jessica Dorsey, Jonathan Kwik, Nema Milaninia, Natalie Nunn, Nivedita Raju, Dan Smith and Fan Yang.

Finally, the authors acknowledge the invaluable editorial work of the SIPRI Editorial Department.

Executive summary

States involved in policy debates on military artificial intelligence (AI) are increasingly expressing concerns about bias in military AI systems—defined here as the phenomenon of a military AI system being unintentionally inclined towards or against certain individuals or groups of people in a way that is systemic and unfair. Yet, these concerns are rarely discussed in depth, much less from a legal lens. To inform policy discussions on how to ensure the lawful use of military AI, this report explores the implications of bias in military AI for compliance with international humanitarian law (IHL). Focusing on bias in AI-enabled autonomous weapon systems (AWS) and AI-enabled decision support systems (DSS) for targeting, the report makes three main findings.

First, bias around gender, ethnicity, disability status, age, socio-economic class, language and culture (among others) can be introduced into a military AI system throughout its development and use. Such demographic-based bias can result in unintended harm through two primary pathways: the misidentification of persons or objects, and failures to detect protected persons and objects.

Second, bias in military AI has implications for compliance with IHL. Misidentifications stemming from bias could lead to violations of the principle of distinction and related obligations to take all feasible precautions to verify that targets are military objectives. Failures to identify protected persons or objects as a result of bias could lead to violations of the principle of proportionality and related obligations to take all feasible precautions in attack and to take constant care to avoid incidental harm to civilians.

Third, bias in military AI cannot be entirely removed, but its consequences can be mitigated through limits and requirements in the development and use of an AI system. In the development phase, measures include dedicated efforts to ensure representative datasets and to review and test for bias in the systems. During use, measures to minimize the risk of bias manifesting as harm could entail operational limits and requirements around target verification processes.

The risks of harm arising from bias in some respects exacerbate existing challenges that the use of military AI poses for compliance with IHL. In light of the importance of addressing bias in military AI, the report recommends that states take the following actions:

• Develop national expertise to account for bias as part of the procurement of military AI systems.
• Elaborate on what IHL requires in terms of addressing bias.
• Use considerations of bias to inform policy discussions around limits and requirements to ensure the lawful use of military AI.

1. Introduction

Many states, especially those with large and resourceful militaries, are increasingly exploring the potential of using artificial intelligence (AI) in targeting decisions. This includes AI-enabled decision support systems (DSS) and AI-enabled autonomous weapon systems (AWS).¹ However, as the interest in adopting AI in the military grows, so does the need to better understand what limits and requirements should guide its use. To this end, states have spent more than a decade discussing the governance of AWS, notably within the framework of the United Nations Convention on Certain Conventional Weapons (CCW).² Over the years, various processes addressing broader issues of military AI have also emerged, including within the UN General Assembly.

Across these discussions, states are increasingly expressing concern about bias in military AI.³ The concern is that military AI systems perpetuate and amplify existing biases, potentially leading to discriminatory and harmful effects, such as inflicting disproportionate harm to specific segments of the population.⁴ In the AWS context, some states have specifically warned that bias could result in inaccurate assessments in the targeting process, and that considerations of bias are therefore key to ensuring compliance with international humanitarian law (IHL).⁵

Yet, the issue of bias in military AI is rarely discussed in depth, nor is it reflected in meeting outcome documents, for example, within the CCW.⁶ This contrasts with the civilian domain, where multinational efforts are well underway to address bias in AI.⁷ One particularly overlooked aspect is the implications of bias in military AI for IHL compliance.⁸ Considering the centrality of IHL compliance in military AI policy debates, the lack of attention paid to the implications of bias represents a gap in efforts to ensure the lawful development and use of military AI.⁹ To fill this gap, SIPRI undertook a research project with the primary aim of exploring what implications bias in military AI has for IHL compliance. This report presents the outcomes of that project.

Key concepts

This report explores the relationship between three key concepts: military AI, bias and IHL. For this report, these concepts are understood, and narrowed down, as follows.

Military AI

AI is a catch-all term that refers to a wide set of computational techniques that allow computers and robots to solve complex, seemingly abstract problems that had previously yielded only to human cognition.¹⁰ AI can be used for various military tasks.¹¹ This report focuses on the use of AI in military applications directly related to the use of force, specifically AI-enabled AWS (AI-AWS) and AI-enabled DSS (AI-DSS) used for targeting. AWS are weapon systems that, once activated, can identify, select and engage targets without human intervention.¹² DSS for targeting are computerized tools that military decision-makers use to inform their targeting decisions.¹³

IHL

IHL is a body of rules setting out requirements, restrictions and prohibitions that must be complied with in armed conflicts.¹⁴ This report focuses on IHL rules regulating the conduct of hostilities, particularly the principles of distinction, proportionality and precautions in attack (box 1). Like many other provisions laid down in IHL treaties, the rules regulating the conduct of hostilities are also reflected in customary IHL and so are binding for all parties to armed conflicts.¹⁵

Bias in military AI

Bias in military AI refers to the phenomenon of a military AI system being inclined towards or against certain individuals or groups of people in a way that is systemic and unfair. Chapter 2 further explains how this kind of bias is understood for the purpose of this report.

Methodology and outline

Research for this report comprised both a literature review and, importantly, engagement with subject-matter experts, primarily via five expert interviews and one in-person workshop. The workshop, held in Stockholm in February 2025, invited 15 experts from government, civil society, industry, academia and the International Committee of the Red Cross (ICRC) to discuss in detail how bias in military AI impacts IHL compliance. The experts were carefully selected to ensure broad representation in terms of geography, gender and areas of expertise. Conducted under the Chatham House rule, the workshop revolved around a series of hypothetical case studies, namely: (a) AI-AWS and AI-DSS used in attacks against persons; (b) AI-AWS and AI-DSS used in attacks against objects; and (c) AI-DSS used to support assessments about incidental civilian injury.

The outcome of this research is presented in the following four chapters. Chapter 2 unpacks the term ‘bias in military AI’, as well as its causes and potential humanitarian consequences. Then, chapter 3 examines the implications of bias in military AI for compliance with the IHL principles of distinction, proportionality, and precautions in attack, as well as the prohibition against adverse distinction. Chapter 4 presents a range of measures to address bias in military AI and ensure compliance with IHL. Finally, chapter 5 outlines key findings and recommendations.

2. Bias in military AI: characterization, causes and concerns

Before exploring how bias in military AI may impact compliance with IHL, it is important to unpack the subject of attention: bias in military AI. ‘Bias’ is a broad term, and without a more detailed understanding of the specifics, it is challenging to sufficiently address it. This chapter describes how ‘bias in military AI’ is understood in this report, identifies sources of bias, and outlines the key humanitarian concerns associated with bias in military AI.

Bias as systemic unfairness

In military AI policy debates, states have expressed different understandings of what ‘bias’ refers to—for example, whether or not bias is an inherently undesired feature of AI systems. The different understandings of what ‘bias’ refers to are also evident in the expert literature.

According to one account, ‘bias’ is a neutral term referring to how an AI system may be inclined towards some characteristics, environments or behaviours in its operation.¹⁶ This inclination can be intentional, desirable and even operationally justifiable.¹⁷ An AWS that is programmed to target a specific type of enemy tank would be an example of a desirable and justified bias towards enemy tanks. However, this inclination can also be undesirable and not operationally justifiable. An example is an AI-DSS that, due to incomplete training data, recognizes light-skinned civilians at higher rates than dark-skinned civilians.

According to other accounts, bias is understood only as the latter—that is, an inherently undesirable inclination towards or against certain individuals or groups of people in a way that is considered unfair.¹⁸

Despite various understandings of what ‘bias’ conceptually refers to, states involved in the military AI policy debates only raise concern about the latter type of bias. For example, Austria, Belgium, Canada, Costa Rica, Germany, Ireland, Mexico, Panama and Uruguay have jointly expressed concern that ‘data-based systems reproduce existing inequalities’, and the United Kingdom has highlighted that ‘the risk of discriminatory outcomes resulting from algorithmic bias or skewed data sets’ is a particular concern with AI-enabled military systems.¹⁹ In other words, when raising the issue of bias in military AI, states’ shared concerns revolve around the risk that the use of military AI may (unintentionally) perpetuate and amplify existing biases around traits such as race or gender, potentially leading to harmful and discriminatory outcomes.

For this reason, this report focuses on the type of bias that states have expressed concern about and a desire to address. ‘Bias in military AI’ in this report therefore refers to the phenomenon of a military AI system being unintentionally inclined towards or against certain individuals or groups of people in a way that is systemic and unfair.²⁰ Referring to bias as a systemic issue indicates that inequitable treatment of or harm to people is not the result of incidental errors but rather the result of structural inequalities.²¹ Unintended bias in military AI can take many forms, usually related to demographics, some of which are illustrated in figure 1.

https://www.sipri.org/sites/default/files/NLWV5347%20Figure%201-01.png

Figure 1. Examples of types of bias that can manifest in military artificial intelligence (AI) systems

Sources of bias in military AI

It is widely recognized that no AI system will ever be entirely free of bias.²² The following outlines three main junctures where bias can be introduced in the development and use of military AI applications.

Bias in society

Bias in society, sometimes referred to as historical or pre-existing bias, represents one of the most fundamental sources of bias in military AI. It refers to the inequalities inherent in societal structures, institutions and practices that inevitably—and often entirely unconsciously—will be mirrored in AI systems.²³

Societal bias typically manifests in the underlying training datasets of military AI systems, in which certain groups of people, environments or behaviours are over- or under-represented.²⁴ In the case of over-representation, characteristics specific to a particular context may be taken as universal representations. For example, the male body could be taken as representative of all body types in AI systems used to assess the physical effects of weapons on people. Or skewed media reports that provide more publicly available images of dark-skinned than white-skinned terrorists on the internet may result in the systems being more inclined to identify black people as threats. Regarding under-representation, datasets may have an existing ‘data deficit’ on certain groups of the population, such as people with disabilities and ethnic minorities.²⁵ Even when datasets are intended to be representative, they may reflect inherent inequalities in society. An additional concern is that AI systems are prone to reinforcing these existing social inequalities or stereotypes because of their ability to identify (and adjust to) hidden patterns in data distribution.

Bias in data processing and algorithm development

Bias in data processing and algorithm development refers to bias arising from the choices and assumptions of the various actors involved in developing military AI systems. Here, bias could manifest through various activities, such as the labelling, modelling and preprocessing of data; the design of algorithms; the learning processes; and the development of training objectives and performance metrics.²⁶ Bias in data processing can arise if programmers emphasize certain outcomes, behaviours or information over others (also referred to as reporting bias). This could lead to a skewed representation in datasets of, for example, enemy characteristics or civilian movements.²⁷ The development of proxy indicators is a good example of how bias can arise in algorithm development. Proxy indicators are indirect ways of measuring something when a direct measurement is not available, and are important components of many machine-learning AI systems. An example is a human’s posture with arms raised as a proxy for surrender. Defining proxies for what targetable people look like is an important element of programming military AI systems. However, the identification of proxies is vulnerable to unintended biases, perceptions and assumptions of programmers.²⁸ This is especially so if choosing proxies for AI systems designed for non-international armed conflicts, where the enemy may not wear easily recognisable uniforms.

Bias in use

Bias in use refers to bias that arises during the deployment of a military AI system, for example, due to new contexts or interactions that were not anticipated during the system’s initial design and training.²⁹ Here, the duration of the conflict combined with the duration of a system’s operation can lead to bias.³⁰ That is, as the conditions on the ground change throughout a conflict, the initial training data no longer matches realities on the ground, leading to a so-called ‘data drift’; for instance, if the number of people with disabilities increases or civilian mobility patterns change.³¹ Bias in use could also result from human–machine interaction during deployment: an AI system that uses positive feedback loops may adopt the preferences of individual users, which could mean that algorithmic outputs come to reflect a specific user’s biased preferences or choices.³²

In closing, bias in military AI can be introduced at various stages during the development and use. A key question is whether such biases could or should have been reasonably anticipated and accounted for in the system’s design or deployment. This is a point relevant for legal compliance that will be tackled in chapter 3.

The primary concerns about bias in military AI

The overarching concern about bias in military AI is that it could expose certain individuals or groups of the population to greater harm during an armed conflict. The pathways for such harm to materialize can be boiled down to two scenarios in the targeting context: misidentification and failure to identify (see figure 2).

https://www.sipri.org/sites/default/files/NLWV5347%20Figure%202.png

Figure 2. Risk pathways through which bias in military artificial intelligence (AI) can manifest as harm

Risk of misidentification

Misidentification refers to instances where an AI system, for example due to bias, misclassifies civilians and civilian objects, or other protected persons and objects, as lawful targets. This risk could arise, for example, in cases where AI systems are used to identify non-uniformed enemy fighters. Here, bias could lead to males hunting or celebrating with gunfire being wrongly classified as lawful targets because the system is trained on data that has characterized enemy behaviour as including groups of young men carrying weapons.³³

Risk of failure to identify

Failure to identify refers to instances where an AI system, due to bias, fails to detect certain persons or objects as present in the target area. This risk could arise, for example, if an AI system is not trained on images of people in wheelchairs and therefore fails to detect this group of the population. Another example is where phone activity is used as a proxy for civilian presence, which could lead the AI system to fail to identify civilian populations with limited internet and phone access. Failures to identify certain civilians and civilian objects could expose them to greater harm if their presence is not accounted for.

A failure to identify certain persons or objects could also lead to situations where an AI system fails to detect lawful targets. For example, if an AI-AWS has only been trained on images of male combatants, it may fail to identify female combatants.³⁴ While such situations may be undesirable from an operational perspective, they do not pose humanitarian risks and are therefore not considered further in this report.

Finally, it is important to note that while both AI-AWS and AI-DSS are prone to harmful bias, how harm manifests may differ. AWS have a direct path between target identification and engagement, so the risks are direct—for example, a false target identification can result in immediate lethal action without human intervention. DSS provide outputs to inform users’ targeting decisions, so the risk is indirect—harm materializes if humans act upon inaccurate proportionality assessments or threat identifications.³⁵ The next chapter explores the ways in which misidentifications and failures to identify caused by bias in both AI-AWS and AI-DSS impact IHL compliance.

3. Legal implications of bias in military AI

No treaty provision or customary rule of IHL expressly refers to ‘bias’, but this absence does not imply that IHL is conceptually or normatively neutral with respect to the kinds of differentiation or discrimination that bias may entail. Crucially, IHL requires parties to an armed conflict to make distinctions. Key distinctions focus on differentiating between combatants, civilians directly participating in hostilities, civilians, and civilian and military objects, and between objects that are military by nature, use, location or purpose and non-military objects (box 2).³⁶ The question is what implications bias in military AI has for making such distinctions.

Informed by existing literature and hypothetical cases involving AI-AWS and AI-DSS discussed during the expert workshop convened by SIPRI, this chapter explores the implications of bias in military AI for compliance with IHL, specifically the rules regulating the conduct of hostilities, including the principles of distinction, proportionality and precautions in attack (box 1). The chapter first examines legal implications in situations where bias in military AI leads to misidentifications—specifically, how misidentifications could undermine compliance with the principle of distinction and related precautionary obligations. Then, considering the risk of failure to identify certain persons and objects, the chapter explores the implications of bias in military AI for compliance with the principle of proportionality and related precautionary obligations. In closing, the chapter considers whether bias in military AI—when understood as discriminatory or unfair treatment—could engage the IHL prohibition against adverse distinction.

Legal implications of misidentification

IHL prohibits directing attacks against unlawful targets and requires all feasible precautions to be taken in an attack to ensure that attacks are directed only at military objectives. Thus, at the core of the principle of distinction is the requirement for precise and reliable target identification. To uphold this principle, parties to an armed conflict must carry out several tasks related to target identification, including distinguishing between civilians and combatants, between active combatants and those who are hors de combat (i.e. combatants who are no longer targetable because they, for example, surrender or are wounded), and between military objects and civilian objects.

However, if AI is used to identify and possibly engage targets, the presence of bias may undermine the ability of parties to make these distinctions.³⁷ While the use of military AI in distinction-related decisions raises challenges, irrespective of bias, the presence of bias could contribute to, and even amplify, existing concerns around how to ensure respect for the principle of distinction.³⁸

The obligation to only direct attacks against lawful military targets

Human targets. If AI is used in attacks against persons, whether through AI-AWS or AI-DSS, bias could have implications for the obligation to only direct attacks against combatants and civilians directly participating in hostilities. This is particularly so due to the risk of harmful biases embedded in proxy indicators used for targeting, such as collective and general identity markers related to age, ethnicity, gender or patterns of behaviour. Though the use of proxies in targeting decisions is a practice (and a related debate) that predates AI, integrating proxies into AI systems could exacerbate existing concerns about the use of general and collective identity markers.³⁹ This is because AI systems might find correlations and inferences in the proxy indicators that the designer did not intend, which risks reproducing errors and amplifying existing inequalities.⁴⁰ This is particularly relevant where proxies are based on assumptions about characteristics associated with combat status in conflicts where the enemy may not be uniformed. Here, communication and behavioural patterns—such as using prepaid phone cards and moving in groups carrying weapons—could be considered proxies, together with other elements, for combat status. However, in some communities, using prepaid phones may be common among civilians and not necessarily an enemy characteristic. Similarly, moving in groups carrying weapons may be a routine cultural practice—such as in ceremonial or religious festivities, or for hunting animals—rather than a sign of coordination for hostile purposes.⁴¹

As an additional concern, bias in the underlying datasets and proxy indicators can also result in persons with disabilities being wrongly identified as threats.⁴² For example, when it comes to behavioural analysis and threat detection, a lack of data about behaviours of persons with psychosocial, intellectual or sensory impairments may lead to interpreting behaviour like a lack of response or shouting as hostile actions.⁴³

Material targets. AI used in attacks against material targets also carries the risk of misidentification due to bias. During the workshop, participants discussed loitering munitions used to identify and possibly engage military weapons storage facilities (military objectives by nature) and temporary command centres (military objectives by use). Both systems relied on predefined criteria, such as shape, layout, architectural styles and building materials, as proxies for targetability. Participants warned that if those involved in the development phase fail to account for regional specificities and variations of buildings, proxy indicators could embed historical or cultural biases, which could lead to misidentification. While these risks applied to both cases, the risks associated with bias were found to be particularly pronounced if AI systems are used in attacks against military objectives by use. Unlike military objectives by nature, their status as lawful targets depends on an assessment of their use, which usually includes human movements in and around these structures. For example, if the system has not been trained on a sufficiently representative, diverse dataset, it may interpret local religious practices around a building as enemy activities and wrongly classify the building as a military objective. As stressed by some experts, the more a distinction assessment depends on soft metrics like human behaviours and characteristics, the more vulnerable it is to bias.⁴⁴

Legal implications. Whether the above-mentioned situations amount to violations of the principle of distinction depends in part on how specific target profiles should be. According to some experts, relying on overly generalized target profiles is difficult to reconcile with the requirements under the principle of distinction to make individualized and context-dependent assessments of the status of persons and objects as lawful targets.⁴⁵ Although IHL ‘does not make explicit that [identity markers] cannot be used as a basis for identifying people as combatants in international armed conflict’, proxies that consciously or unconsciously reflect cultural stereotypes or oversimplified human behaviours are unlikely to be sufficient alone to capture the dynamic nature of some military objectives, as required by the principle of distinction.⁴⁶ And arguably, if not used with caution, the use of proxy indicators may undermine compliance with the principle of distinction.⁴⁷

At the same time, others stress that IHL in fact offers parties to a conflict broad discretion in developing target profiles.⁴⁸ This discretion complicates the evaluation of whether proxies for target status, such as gender or age, or unintended proxies related to target status, are arbitrary, and thus whether targeting practices based on these proxies amount to unlawful distinctions (box 2). However, if a targeting criterion is unreasonable and results in foreseeable and systematic errors, such practices would fall outside the scope of lawful discretion and may constitute violations of IHL.

The risk of misidentification due to bias is present for both AI-AWS and AI-DSS. However, these systems’ distinct forms of human–machine interaction mean that the way in which misidentification would manifest as harm and possibly undermine distinction obligations differs. In the case of AI-AWS, misidentification would result in direct harm to protected persons or objects. For AI-DSS, harm results only if users act on potentially flawed targeting recommendations. The central issue, therefore, is how AI-DSS outputs impact and shape users’ distinction-related assessments. Importantly, AI-generated outputs only provide probabilistic assessments (i.e. statistical correlations) as to whether an object or person is targetable, which cannot replace the human context-dependent judgement required by IHL for target identification.⁴⁹

The obligation to recognize surrender

The principle of distinction prohibits attacks against combatants who surrender or are wounded, that is, persons hors de combat.⁵⁰ The ability to comply with this obligation has long been a concern in the context of AWS because compliance is associated with a need for constant human supervision during an attack.⁵¹ When adding risks of misidentification flowing from bias, the concern becomes more significant, especially with regard to recognizing surrender. In the hypothetical case study discussed at the workshop, an anti-personnel loitering munition was trained to detect surrender motions (in the shape of arms held above the head) and was programmed not to engage if it detected such gestures. However, several participants emphasized that surrendering gestures are likely to be context- and culture-dependent. If an AI-AWS or AI-DSS is trained to recognize surrender only through the gesture of raising both arms above the head, it may risk overlooking alternative and culturally specific signs of surrender, potentially leading to the targeting of unlawful targets. Moreover, because acts of surrender represent a very specific behaviour that does not necessarily occur often during conflict, datasets often lack sufficient examples of hors de combat individuals, leading to statistical bias.⁵²

Workshop participants highlighted that bias in data curation—stemming, for example, from a lack of cultural awareness—can significantly undermine the lawful use of AI-AWS. It was broadly agreed that AI-AWS, which are not designed to ‘recognize’ culturally specific ways of surrendering, could not be used to attack humans in accordance with the principle of distinction—if anti-personnel AWS should be used at all.⁵³

The obligation to take all feasible precautions in attack

Whether a misidentification flowing from bias in a military AI system amounts to a violation of the principle of distinction depends on, among other factors, the fulfilment of the obligation to take all feasible precautions in attack. The obligation to take all feasible precautions in attack serves to ensure compliance with the principle of distinction through positive actions, including the overarching obligation of taking constant care in the conduct of military operations. The temporal scope of this obligation is not necessarily limited to the moment of attack but extends throughout the planning and decision-making phases of military operations.⁵⁴ The obligation imposes a continuing duty on those deciding and planning on an attack to mitigate civilian harm by undertaking precautionary measures in advance of initiating or authorizing an attack.⁵⁵ According to several authorities, the obligation to ensure respect for IHL must be carried out with due diligence—defined as a standard of care or parameter used to assess states’ implementation and compliance with IHL (although what the standard entails and requires from states is disputed in certain respects). The obligation to take all feasible precautions in attack arguably includes due diligence during the design of AI-AWS and AI-DSS, such as ensuring sufficiently representative training datasets and proxy indicators.⁵⁶ Negligent data selection and curation, such as relying on data only collected from Western media outlets, could amount to a failure to satisfy due diligence obligations.

Importantly, the principle of precautions in attack also requires parties to a conflict to do everything feasible to verify that targets are military and not civilian.⁵⁷ This raises questions about what standard of certainty is required when using AI to conduct target verification, and what corroborative information is necessary to support the system’s outputs.⁵⁸ The speed and scale at which AI-DSS analyse information and generate targeting recommendations could lead to verification processes becoming largely procedural or symbolic, described in some accounts as akin to ‘rubber stamping’.⁵⁹ Automation bias—the tendency to over-rely on AI-generated outputs—may further compromise the ability to verify targets, as it may discourage critical cross-checking.⁶⁰ Thus, if users act on flawed target recommendations without considering the possibility of bias and taking further steps to verify the target, this will be a breach of the obligation to take all feasible precautions in attack, rather than a violation of the principle of distinction.

Importantly, a failure to meet precautionary obligations does not require proof that the harm was intentional; a negligent failure to take reasonable steps to avoid such outcomes may be enough to incur a violation.⁶¹ This reinforces the importance of embedding bias-awareness and mitigation as part of the obligation to take constant care in military operations. In particular, states should clarify whether precautions in attacks require multi-tiered and iterative target verification procedures that include deliberate checks for bias.

Legal implications of failures to identify

The principle of proportionality requires parties to an armed conflict to assess and balance the anticipated military advantage associated with a military objective and the expected incidental harm to civilians.⁶² This assessment depends on the ability to distinguish between military and civilian objects and the ability to reasonably foresee and assess the potential incidental harm.⁶³ According to some, the use of AI-DSS contains the potential to support such proportionality-related assessments.⁶⁴ While the use of AI to support proportionality assessments raise a series of legal challenges irrespective of bias, this section considers the specific implications of bias for compliance with IHL in such cases.⁶⁵ As it is widely recognized that the contextual and evaluative nature of proportionality assessments renders it legally and technically unfeasible to delegate proportionality assessments to AWS, the section focuses on a limited range of proportionality-related tasks that AI-DSS could be used for.⁶⁶

The obligation to ensure that attacks are proportionate

Workshop participants discussed the use of an AI-DSS battle management system to provide assessments of the operational environment, including the presence of civilians and civilian objects in the vicinity of an attack. The system was also used to recommend the choice of weapons based on weapon radius, angle and payload, as well as data on civilian presence. Many participants stressed that the system in the scenario was prone to bias, especially if it relied on machine learning. For example, if the AI-DSS is not trained on a dataset that sufficiently reflects the complexity and diversity of the local population, including in terms of ethnicity, socio-economic class or disability status, or if the system is not iteratively reflective of the changing and evolving nature of the battlespace, it could draw correlations that disproportionately affect some people or objects and fail to recognize others. Relatedly, the ability to account for the composition of the population is also important in light of gender analysis of the impacts of military operations, which has revealed significant variation in civilian harm—for example, certain weapons’ effects on different segments of the population, such as children or pregnant people.⁶⁷ In addition, where an AI-DSS relies on pre-conflict data, it could fail to sufficiently account for changes in the composition of the civilian population and changes in mobility patterns. For example, the number of people with disabilities has been shown to increase from around 16 per cent during peacetime to up to 30 per cent during conflict.⁶⁸ AI-DSS that fail to account for these shifts may generate erroneous assessments and predictions, potentially skewing the commander’s analysis in a proportionality assessment.⁶⁹

Relying on information that fails to account for all protected persons or objects will likely underestimate the expected harm or lead to flawed weapons recommendations, thus raising the serious risk of inflicting disproportionate harm in attacks.⁷⁰ That is, if bias is not accounted for in a proportionality assessment, bias in AI-DSS could lead to violations of the principle of proportionality. However, the principle of proportionality requires users to consider expected incidental harm to civilians and civilian objects in relation to the anticipated military advantage. Thus, compliance with this rule does not require absolute certainty about the extent or occurrence of civilian harm. Instead, it is based on a reasonable assessment of what could be foreseen at the time of the attack, allowing for the possibility of unintended or unforeseen harm. This raises the question of when the presence of bias in AI-DSS could, or should, have been reasonably foreseen and accounted for.

The obligation to take constant care in military operations to spare the civilian population, civilians and civilian objects

IHL requires parties to a conflict to take all feasible precautions in attack and constant care in military operations to avoid or at least minimize civilian harm.⁷¹ Whether harm caused by biased outputs of an AI-DSS constitutes a violation of the principle of proportionality will hinge, in part, on whether these precautionary obligations were met. For example, the exclusion of certain population groups from AI-DSS datasets would demonstrate a failure to comply with these precautionary obligations because they require that parties to the conflict take into account the diversity of the whole civilian population, including differences in gender, age, ethnicity and disability, to ensure that all groups of the population receive equal protection.⁷² This entails positive obligations to ensure that the protection of civilians is implemented in a manner that is inclusive, non-discriminatory, and attentive and responsive to the varied needs and vulnerabilities of different population groups affected by armed conflict. Importantly, as mentioned above, a failure to meet precautionary obligations does not require proof that the harm was intentional; simply a negligent failure to take reasonable steps to avoid such outcomes—in these cases, a failure to reasonably assess the expected incidental harm—may be enough to signal a violation.⁷³

The prohibition against adverse distinction

The IHL rule that comes the closest to explicitly addressing bias is the prohibition against ‘adverse distinction’. This rule—which applies in both international and non-international armed conflicts and is found in both IHL treaty and customary law—prohibits adverse distinction in the treatment of persons based on race, colour, religion, sex, political or other opinion, birth or other status, or ‘any other similar criteria’.⁷⁴ Initially developed in international human rights law (IHRL), the rule can be understood as a prohibition against discrimination and unequal treatment.⁷⁵ An analysis of human rights instruments and case law suggests that the prohibition against adverse distinction applies to situations of direct discrimination (e.g. depriving some prisoners of war of food based on their sexual orientation) and indirect discrimination—that is, adverse effects that may arise from uniform treatment of persons in relatively different situations (e.g. serving identical meals to all prisoners without regard for dietary restrictions based on religious beliefs).⁷⁶ The thresholds for assessing whether an act or result amounts to adverse distinction are determined by whether the distinctions are arbitrary according to criteria drawn by IHRL (box 2).

However, while it is well-established that the prohibition against adverse distinction applies to the treatment of persons in the power of a party (such as prisoners of war), it is unclear whether it also applies to the conduct of hostilities.⁷⁷ Some states and IHL experts have explicitly expressed the view that the prohibition applies to targeting, but the issue remains subject to debate and largely unaddressed in the literature and among states.⁷⁸ If states accepted that the prohibition against adverse distinction does apply to targeting, it could have implications for compliance with IHL in the context of military AI.⁷⁹ For example, if an AI-DSS systematically overlooks certain minorities when assessing the presence of civilians, the use of such system might not only impact compliance with the principle of proportionality, but also the prohibition against adverse distinction. Additionally, if applicable, the prohibition would provide further guidance for states on how to fulfil their obligations regarding distinction, proportionality and precautions in attack, especially when using military AI in targeting. However, even if states do not consider that the prohibition against adverse distinction applies to targeting, they arguably still have a duty to prevent de facto discriminatory effects in targeting, a duty that follows from their non-debatable obligations to take constant care in military operations and to prevent or at least minimize civilian harm. In any case, whether the prohibition against adverse distinction applies to targeting remains an open question to be further elaborated by states, including the standards of behaviour required of parties to a conflict to prevent discriminatory practice and effects during the conduct of hostilities, especially where these arise from the use of military AI.

Bias in military AI matters for IHL compliance

Considerations of bias in military AI are relevant for compliance with IHL rules regulating the conduct of hostilities, because bias can affect system performance in ways that impact target identification and the protection of civilians and civilian objects.

In the context of AI-AWS, the risks of misidentification flowing from bias are particularly acute due to their capacity for direct kinetic engagement of targets without real-time human intervention. Such target engagement could impair compliance with the principle of distinction (where bias is a contributing cause of unlawful outcomes) and the obligation to take all feasible precautions in an attack (where a failure to take bias into account could be considered a failure to take constant care in military operations). In the context of AI-DSS, both the risks of misidentification and failure to identify protected persons and objects are acute insofar as these systems are used to support and shape decisions around distinction and proportionality. User over-reliance on potentially flawed outputs could lead to violations of the principles of distinction and proportionality, as well as the obligation to take all feasible precautions in attack, including the obligation to do everything feasible to verify targets.

Bias in military AI also potentially violates the prohibition against adverse distinction because it could lead to discriminatory conduct and effects. However, whether this prohibition applies to targeting is an open question that deserves closer scrutiny.

4. Measures to address bias in military AI and ensure respect for IHL

Consideration of bias in military AI is important to help ensure compliance with IHL, including the principles of distinction, proportionality and precautions in attack. A key question for policymakers is how such considerations translate to specific measures, potentially to be embedded in military AI governance instruments.

Since it is impossible to entirely ‘debias’ AI systems from unintended, potentially harmful biases, this chapter explores limits and requirements intended to address bias. ‘Addressing’ bias spans from measures to detect and correct biases, as well as measures to limit potential harm.

While none of the suggested measures is ‘unique’ to bias (being generally considered important for ensuring lawful use of military AI), specific attention to bias provides specificity and nuance as to why such measures will strengthen compliance with IHL and what they should entail.

Representative and context-specific data

A key measure to address bias in AI-AWS and AI-DSS is ensuring that the underlying datasets sufficiently represent and align with the intended operational environment.⁸⁰ Representative and context-specific datasets that, for example, include local religious minorities, family structures and dialects are associated with a lower risk of misidentifications and failures to detect local minorities.⁸¹ Context-specific, and therefore smaller, datasets are also thought to make it easier for users to detect and account for biases.⁸²

Ensuring representative datasets requires developers of military AI systems to consider the intended use from the outset, at the development, testing and programming phases. This necessitates collecting data specific to the intended environment of use and data based on the most recent information about the conflict, rather than extrapolating from large, general datasets. Weapons reviews can serve as essential junctures to check and detect possible bias in the system’s datasets and ensure that the datasets are sufficiently representative of the intended environment of use.⁸³ Weapons reviews could also incorporate systematic bias detection—for example, by testing for discriminatory outputs across demographic variables or by including requirements for multi-context training datasets.

However, ensuring that datasets are representative and context-specific raises fundamental questions about the trade-off between data quality and respect for human rights, such as the right to privacy. That is, the more contextualized the data, the greater the need to collect data from local populations, potentially requiring expanded data collection and surveillance measures, which may come at the cost of increased intrusions into individual privacy. Indeed, some experts have warned that surveillance is the flipside of contextualized data.⁸⁴

The need for representative and context-specific datasets also raises the question of whether those developing the systems can be expected to foresee the intended environment in which a system may be used and to know the relevant contextual details. This is particularly challenging because many military AI systems are developed on the basis of foundation models, and so are built and trained by private actors.⁸⁵

Data transparency

Data transparency refers to the ability for users to access, or at least be informed about, the data on which a system was built and trained. It is considered crucial in addressing bias and ensuring compliance with IHL in the context of military AI.⁸⁶ First, insight into an AI system’s training data will help users understand on what basis the system produces target recommendations, including whether it contains any bias—for example, whether the system is skewed towards certain characteristics or uses data that potentially does not match the environment of use. To this end, according to consulted experts, operators should be informed about what data was represented in the training datasets, how the data was labelled, how the system was trained and how the system performed during testing. Second, data transparency will make it easier for users to investigate harmful events because it could help explain the causes and possibly establish responsibility.⁸⁷

However, ensuring data transparency and oversight faces both technical and organizational challenges. From a technical perspective, audit tools for interrogating and evaluating complex processes are key transparency measures that can be deployed as part of the development process to examine model design (usually internally) or after the system has been deployed.⁸⁸ However, some technical experts consider it impossible to sufficiently audit and interpret military AI systems that are based on machine learning, given the opaque nature of such systems, often described as operating within a ‘black box’.⁸⁹ From an organizational perspective, the industrial secrecy of private companies poses another hindrance to promoting data transparency. While some consulted governmental representatives said that they would never procure a system if they could not access the training data from the developers, it is unclear whether that attitude is shared among states.⁹⁰

Human control and judgement during an attack

Considerations of bias in military AI with a view to strengthening respect for IHL cannot be addressed solely through ‘technical fixes’. Retaining human control and judgement during an attack is one of the most critical measures for addressing bias and ensuring respect for IHL.⁹¹ The different capabilities and functions of AI-AWS and AI-DSS mean that human control and judgement are achieved differently in each case.

For AI-AWS, human control and judgement are generally considered to be achieved through operational limits. And while the need to place limits on the use of AWS is already widely recognized as a key measure to ensure compliance with IHL, the risks flowing from bias emphasize this need. For example, most of the consulted experts argued that AI-AWS should not be used in populated environments because the more dynamic an environment is, the harder it is to ensure a sufficiently representative dataset.⁹² Also, most of the consulted experts expressed that limiting the use of AI-AWS to attacks against military objectives by nature would be key to minimizing the consequences of bias. That would mean not using AI-AWS in attacks against persons or military objectives by use, location or purpose. The underlying rationale is that using AI-AWS in attacks against persons or objects whose status as lawful targets depends on assessments of human behaviour (e.g. hostile activity, gestures of surrender, or patterns of behaviour) is particularly vulnerable to harm materializing as a result of bias.⁹³

For AI-DSS, which, in contrast to AI-AWS, operate under constant human oversight, human control and judgement are retained through requirements to avoid that users become passive approvers of the systems’ outputs. Here, ensuring users’ ability to review, challenge and validate AI recommendations is considered essential for spotting biases and minimizing the risk of relying on recommendations that, for instance, fail to take into account the presence of certain parts of the population.⁹⁴ This would require military organizations to have multiple layers of verification in place, including cross-checking with other sources, so users do not rely solely on the output of an AI-DSS.⁹⁵ The need to cross-check sources is considered an important tool not only for identifying and mitigating bias in the systems, but also for addressing the cognitive biases of the users. Cognitive biases, such as automation or confirmation bias, are considered a ‘risk magnifier’ because they can result in user over-reliance on potentially biased recommendations of the AI systems. Cross-referencing output is already, regardless of AI, considered good practice to mitigate cognitive bias.⁹⁶ Importantly, the speed and scale at which an AI’s outputs are produced must not undermine the conditions needed for its users to cross-check and validate the outputs.

Institutional measures

Institutional measures within military organizations are also considered essential to minimize and address bias in AI-AWS and AI-DSS and uphold IHL obligations.⁹⁷ To this end, the emphasis is often on training to detect bias and on organizational diversity.

Training to detect bias

Several states and subject-matter experts have emphasized the importance of strengthening internal capacity on bias in military AI. The better the understanding military organizations have of specific types of bias and their associated risks, the better equipped they will be to detect bias.⁹⁸ This would entail that military organizations first invest in developing their understanding of bias in military AI. Then, based on a detailed understanding, military personnel should receive training on how to detect bias.⁹⁹ With a particular focus on unintended bias, such training should raise awareness about the types and sources of bias in military AI systems, as well as situations particularly vulnerable to bias. Military personnel across the development and use of military AI should receive such training, including those involved in the weapons review process, as well as end-users such as operators and commanders. However, whether requirements to spot and detect biases overburden military AI users, especially if such systems are used in a fast-paced and dynamic operational environment, remains an open question.¹⁰⁰

Organizational diversity

Another institutional measure to address bias in military AI is to ensure diversity in the teams involved in the development, testing and programming phases.¹⁰¹ For example, diversity among the people who collect and label the data, including those who decide on proxy indicators, can be critical to minimizing bias in military AI systems.¹⁰² This could also include incorporating, at the data-processing and algorithm development stages, gender and cultural advisers with specific knowledge about the intended context of use.¹⁰³ Teams should strive to include developers of diverse genders, ethnicities, religions and socio-economic backgrounds. However, it is essential to emphasize that to prevent diversity exercises from becoming purely performative, proactive measures are necessary to challenge power dynamics and institutional barriers that often hinder certain demographic groups from driving change within their organizations.

The role of bias mitigation measures in governance frameworks

In closing, elaborating on measures to address bias could help states identify elements of governance frameworks related to military AI. In the context of discussions within the GGE on LAWS, for example, Ireland has already expressed that a ‘future normative framework should include positive obligations and commitments with regards to bias in the use of AWS to support compliance with IHL’.¹⁰⁴

Besides identifying concrete measures to address bias in military AI, a subsequent question is whether such measures are legally required or just elements of good practice. For example, what positive actions, in terms of bias detection and mitigation, flow from states’ obligations to take all feasible precautions in attack? Or, more broadly, what positive actions to address bias flow from states’ obligations to ensure respect for IHL? Here, it is worth recalling that the obligation to ensure respect for IHL is generally seen as as requiring states to ensure that IHL is implemented and applied at the national level, which could entail a range of preventive and supervisory measures, including education, training and legal advice.¹⁰⁵

5. Key findings and recommendations

The previous chapters explored how bias in military AI, specifically AI-AWS and AI-DSS used for targeting, manifests and impacts compliance with IHL. This chapter summarizes the key findings and offers recommendations to states involved in the policy debates on military AI, regarding how they should respond to the challenges posed by bias and ensure lawful use of military AI.

Key findings about bias in military AI

Demographic-based bias in military AI poses humanitarian risks and can result in unintended harm

Unintended and undesirable bias in military AI systems can be introduced at multiple junctures during the development and use of a military AI system, for example through data collection and processing, algorithm development and training. The types of biases that pose particular risks in the military context include biases around gender, ethnicity, age, culture, socio-economic class and disability status. The humanitarian risks flowing from biases in military AI can result from two general risk pathways: (a) misidentification of persons or objects, and (b) failures to detect protected persons and objects.

Bias in military AI can lead to discriminatory targeting practices and potentially unlawful attacks

The risks of misidentification and failures to detect protected persons or objects as a result of bias in military AI used for targeting can manifest as discriminatory practices and harmful outcomes. These could ultimately lead to violations of the principles of distinction, proportionality and precautions in attack.

The risk of misidentification, where non-threats are incorrectly identified as threats, is particularly acute for AI-AWS due to their capacity for direct kinetic engagement of targets without real-time human intervention. However, the risk also arises in the use of AI-DSS, insofar as users act on flawed target recommendations. Both situations could impair compliance with the principle of distinction where bias is a contributing cause of unlawful outcomes.

The risk of failure to identify protected persons and objects is particularly acute for AI-DSS. Users acting on the outputs of an AI-DSS that fails to account for all groups of the population could lead to an underestimation of expected civilian harm, potentially distorting proportionality assessments.

However, whether misidentifications or failures to identify due to bias amount to violations of the principle of distinction or proportionality largely depends on whether the presence of bias should or could have been reasonably foreseen.

That said, even if not intended, bias that could have been reasonably foreseen could amount to a violation of the obligation to take all feasible precautions in attack. This obligation requires constant care in military operations through positive actions—that parties to a conflict do everything feasible to verify that the objectives to be attacked are military objectives and to minimize incidental harm to civilians and civilian objects. Importantly, a violation of precautionary obligations does not require proof that the harm was intentional; a negligent failure to take reasonable steps to avoid the harm may be enough to constitute a violation.

Bias in military AI cannot be entirely removed, but it can be addressed through limits and requirements in development and use

To ensure respect for IHL in the development and use of military AI, states should take measures to address bias in these systems. ‘Addressing’ bias spans from measures to detect and correct biases to measures to minimize potential harm. In the development phase, this includes dedicated efforts to ensure that datasets are representative, context-specific and transparent, and that bias detection is an integral part of testing and evaluating the systems. During the use phase, a key measure to address bias in AI-AWS would be to restrict the use to situations that are less vulnerable to harmful biases. This could include limiting the use to situations that do not depend on assessments of qualitative metrics, such as gestures of surrender, hostile behaviours or context-dependent usages of objects. For AI-DSS, measures during use revolve around ensuring that users have sufficient time and resources to review and cross-check the system’s outputs.

Recommendations for states

Develop national expertise to account for bias in military AI systems as part of the procurement of military AI systems

Understanding how bias arise and manifest in different applications of military AI is a critical starting point for addressing the risks associated with it. States seeking to acquire military AI should take concrete steps to expand their understanding of the sources and types of bias in military AI and streamline this knowledge throughout their military organizations. Specifically, this could include educating developers, advisers and procurers on the various sources and types of biases in AI systems, as well as on the ways in which such bias could result in harm. It could also include recruiting specialists in addressing bias to existing development and procurement teams. In addition, states could ensure that mapping sources of bias becomes an established routine at the development and procurement stage, for example, during weapons reviews.

Elaborate on what IHL requires in terms of addressing bias

To ensure respect for IHL in the development and use of military AI, states should articulate their views on what IHL requires with regard to addressing bias. It would be particularly helpful for states to elaborate on what the obligations to ensure respect for IHL and to take constant care in military operations require, both in the development and use phases. This could include specifying requirements for reviewing and testing AI systems for biases, as well as how users should account for bias during target verification and assessments of expected incidental harm to civilians.

In addition, states could clarify national interpretations of the prohibition against adverse distinction and its application to targeting. Such clarification could help foster a more precise understanding of what the principles of distinction, proportionality and precautions in attack require with respect to non-discrimination. It could also support efforts to address systemic targeting errors and prevent the inaccurate or biased representations of civilian harm in AI-supported targeting processes.

Include considerations of bias in efforts to identify what limits and requirements are needed to ensure the lawful use of military AI

States involved in military AI policy debates should make bias considerations a key element in discussions on how to ensure the lawful use of military AI. In the context of AWS, states could, for example, explore how the risks of bias can be used to identify limits needed on where and against what/whom AWS can be used to ensure compliance with the principles of distinction and precautions in attack. In the context of AI-DSS, states could explore how the risks of bias can be used to specify what is required from users to validate and cross-check sources in targeting decisions to ensure compliance with the principles of distinction, proportionality and precautions in attack.

Endnotes

1. 1 See e.g. Boulanin, V. and Verbruggen, M., Mapping the Development of Autonomy in Weapon Systems (SIPRI: Stockholm, Nov. 2017) pp. 36–54; Nadibaidze, A., Bode, I. and Zhang, Q., AI in Military Decision Support Systems: A Review of Developments and Debates (Centre for War Studies: Odense, Nov. 2024); Israel Defense Forces (IDF), ‘The IDF’s use of data technologies in intelligence processing’, IDF press release, 18 June 2024; and Bendett, S., ‘Roles and implications of AI in the Russian–Ukrainian conflict’, Center for a New American Security, 20 July 2023.
2. 2 Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons which may be Deemed to be Excessively Injurious or to have Indiscriminate Effects (CCW Convention, or ‘Inhumane Weapons’ Convention), opened for signature 10 Apr. 1981, entered into force 2 Dec. 1983.
3. 3 See e.g. CCW Convention, Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems (CCW GGE on LAWS), Report of the 2021 session, CCW/GGE.1/2021/3, 22 Feb. 2022; CCW GGE on LAWS, ‘Addressing bias in autonomous weapons’, Working paper submitted by Austria et al., CCW/GGE.1/2024/WP.5, 8 Mar. 2024; CCW GGE on LAWS, ‘Proposal: Roadmap towards new protocol on autonomous weapons systems’, Working paper submitted by Argentina et al., CCW/GGE.1/2022/WP.3, 8 Aug. 2022; United Nations, General Assembly, ‘Lethal autonomous weapons systems’, Report of the Secretary-General, A/79/88, 1 July 2024; and United Nations, General Assembly, First Committee, ‘Artificial intelligence in the military domain and its implications for international peace and security’, Draft resolution A/C.1/79/L.43, 16 Oct. 2024.
4. 4 For an overview of how existing biases, especially those related to gender, impact compliance with international humanitarian law (IHL), see International Committee of the Red Cross (ICRC), Gendered Impacts of Armed Conflict and Implications for the Application of International Humanitarian Law (ICRC: Morges, June 2022).
5. 5 CCW GGE on LAWS, CCW/GGE.1/2024/WP.5 (note 3); and CCW GGE on LAWS, First session, Seventh meeting, 7 Mar. 2024, UN Web TV, Intervention by Ireland, 00:45:00, and Intervention by Canada, 01:10:00.
6. 6 Mohan, S. and Cho, D., ‘Gender and lethal autonomous weapons’, UN Institute for Disarmament Research (UNIDIR) Fact sheet, Aug. 2024.
7. 7 See e.g. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Official Journal of the European Union, L series, 12 July 2024, Articles 10 and 15.
8. 8 See e.g. Chandler, K., Does Military AI Have Gender? Understanding Bias and Promoting Ethical Approaches in Military Applications of AI (UNIDIR: Geneva, 2021); and Bhila, I., ‘Putting algorithmic bias on top of the agenda in the discussions on autonomous weapons systems’, Digital War, vol. 5, no. 3 (2024).
9. 9 See e.g. CCW GGE on LAWS, Report of the 2023 session, CCW/GGE.1/2023/2, 24 May 2023; CCW GGE on LAWS, ‘Implementing IHL in the use of autonomy in weapon systems’, Working paper submitted by the United States, CCW/GGE.1/2019/ WP.5, 28 Mar. 2019; CCW GGE on LAWS, ‘United Kingdom proposal for a GGE document on the application of IHL to emerging technologies in the area of lethal autonomous weapon systems (LAWS)’, Proposal submitted by the United Kingdom, Mar. 2022; CCW GGE on LAWS, ‘Joint submission on possible consensus recommendations in relation to the clarification, consideration and development of aspects of the normative and operational framework on emerging technologies in the area of lethal autonomous weapon system’, Submission by Austria et al., 2021; and ICRC, ‘ICRC position on autonomous weapon systems’, 12 May 2021.
10. 10 See the detailed definition in Boulanin, V., ‘Artificial intelligence: A primer’, ed. V. Boulanin, The Impact of Artificial Intelligence on Strategic Stability and Nuclear Risk, Volume I, Euro-Atlantic Perspectives (SIPRI: Stockholm, May 2019), pp. 13–25.
11. 11 Geiß, R. and Lahmann, H. (eds), Research Handbook on Warfare and Artificial Intelligence (Edward Elgar: London, 2024).
12. 12 Boulanin and Verbruggen (note 1), pp. 36–54; and ICRC, ‘ICRC position on autonomous weapon systems’ (note 9).
13. 13 ICRC and Geneva Academy, Artificial Intelligence and Related Technologies in Military Decision-Making on the Use of Force in Armed Conflicts, Expert Consultation Report (ICRC and Geneva Academy: Geneva, 2024), p. 9; Gadepally, V. N. et al., ‘Recommender systems for the department of defense and intelligence community’, Lincoln Laboratory Journal, vol. 22, no. 1 (2016); Klonowska, K., ‘Article 36: Review of AI decision-support systems and other emerging technologies of warfare’, eds T. D. Gill et al., Yearbook of International Humanitarian Law 2020 (Asser Press: The Hague, 2021); and Nadibaidze, Bode and Zhang (note 1).
14. 14 Among the key sources of IHL are the four 1949 Geneva Conventions (GCs) and two additional protocols (APs): Geneva Convention (I) for the Amelioration of the Condition of the Wounded and Sick in Armed Forces in the Field (GC I); Convention (II) for the Amelioration of the Condition of the Wounded, Sick and Shipwrecked Members of Armed Forces at Sea (GC II); Convention (III) Relative to the Treatment of Prisoners of War (GC III); Convention (IV) Relative to the Protection of Civilian Persons in Time of War (GC IV), opened for signature 12 Aug. 1949, entered into force 21 Oct. 1950; Protocol I Additional to the 1949 Geneva Conventions, and relating to the Protection of Victims of International Armed Conflicts (AP I); and Protocol II Additional to the 1949 Geneva Conventions, and relating to the Protection of Victims of Non- International Armed Conflicts (AP II), opened for signature 12 Dec. 1977, entered into force 7 Dec. 1978.
15. 15 In 2005 the ICRC compiled and published these rules in a widely accepted and referenced study on customary IHL: Henckaerts, J. and Doswald-Beck, L., Customary International Humanitarian Law, Volume 1: Rules (Cambridge University Press: Cambridge, 2005). This study is now available online as the ICRC, Customary International Humanitarian Law (CIHL) Database <https://ihl-databases.icrc.org/customary-ihl/eng/docs/home&gt;.
16. 16 See e.g. Crawford, K., Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence (Yale University Press: New Haven, CT, 2021); and Danks, D. and London, A. J., ‘Algorithmic bias in autonomous systems’, ed. C. Sierra, Proceedings of the 26th International Joint Conference on Artificial Intelligence (AAAI Press: Melbourne, 2017).
17. 17 Danks and London (note 16); Kwik, J., Lawfully Using Autonomous Weapon Technologies (TMC Asser Press: The Hague, 2024); UK House of Lords, Select Committee on Artificial Intelligence, ‘AI in the UK: Ready, willing, and able?’, Report of Session 2017–19, HL Paper No. 100, 16 Apr. 2018, p. 42; and CCW GGE on LAWS, First session, Sixth Meeting, 6 Mar. 2024, UN Web TV, Intervention by the USA, 01:59:00.
18. 18 See e.g. Ziosi, M., Watson, D. and Floridi, L., ‘A genealogical approach to algorithmic bias’, Minds and Machines, vol. 34, no. 9 (2024), p. 2; Friedman, B. and Nissenbaum, H., ‘Bias in computer systems’, ACM Transactions on Information Systems, vol. 14, no. 3 (1996); Ferrera, E., ‘Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies’, Sci, vol. 6, no. 1 (Mar. 2024); Coeckelbergh, M., AI Ethics (MIT Press: Cambridge, MA, 2020); Buolamwini, J. and Gebru, T., ‘Gender shades: Intersectional accuracy disparities in commercial gender classification’, Proceedings of Machine Learning Research, Volume 81: Conference on Fairness, Accountability and Transparency, 23–24 February, New York (ML Research Press: 2018); and CCW GGE on Laws, First session, Sixth Meeting (note 17), Intervention by Austria, 02:23:00.
19. 19 See, respectively, CCW GGE on LAWS, CCW/GGE.1/2024/WP.5 (note 3), para. 4; and British Ministry of Defence, Ambitious, Safe, Responsible: Our Approach to the Delivery of AI-enabled Capability in Defence, Policy paper, 15 June 2022, p. 11. See also CCW GGE on LAWS, CCW/GGE.1/2022/WP.3 (note 3), para. 17; CCW GGE on LAWS, Working paper submitted by Bulgaria et al., CCW/GGE.1/2024/WP.3, 4 Mar. 2024; and US Department of Defense (DOD), ‘Autonomy in weapon systems’, DOD Directive no. 3000.09, 25 Jan. 2023, para. 1.2(f).
20. 20 Blanchard, A. and Bruun, L., ‘Bias in military artificial intelligence’, SIPRI Background Paper,, Dec. 2024); Kwik (note 17); Ziosi, Watson and Floridi (note 18); Friedman and Nissenbaum (note 18); and Ferrera (note 18).
21. 21 Noble, S. U., Algorithms of Oppression: How Search Engines Reinforce Racism (NYU Press: New York, 2018).
22. 22 UK House of Lords, Select Committee on Artificial Intelligence (note 17), pp. 41–42; Views expressed during the SIPRI expert workshop, Stockholm, 3–4 Feb. 2025; and Holland Michel, A., ‘The machine got it wrong? Uncertainties, assumptions, and biases in military AI’, Just Security, 13 May 2024.
23. 23 Friedman and Nissenbaum (note 18); Suresh, H. and Guttag, J., ‘A framework for understanding sources of harm throughout the machine learning life cycle’, Proceedings of 2021 ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EEAMO ’21) (Association for Computing Machinery: New York, 2021); Bhila (note 8); and Holland Michel, A., Decisions, Decisions, Decisions: Computation and Artificial Intelligence in Military Decision-making (ICRC: Geneva, 2024).
24. 24 Milaninia, N., ‘Biases in machine learning models and big data analytics: The international criminal and humanitarian law implications’, International Review of the Red Cross, no. 913 (Apr. 2020); Bartoletti, I. and Xenidis, R., Study on the Impact of Artificial Intelligence Systems, Their Potential for Promoting Equality, Including Gender Equality, and the Risks They May Cause in Relation to Non-discrimination (Council of Europe: Strasbourg, 2023); and Ferrera (note 18).
25. 25 UK House of Lords, Select Committee on Artificial Intelligence (note 17), p. 42.
26. 26 Blanchard and Bruun (note 20); Schelter, S. and Stoyanovich, J., ‘Taming technical bias in machine learning pipelines’, Bulletin of the Technical Committee on Data Engineering, vol. 43, no. 4 (2020); UNIDIR, ‘Algorithmic bias and the weaponization of increasingly autonomous technologies: A primer’, UNIDIR Resources no. 9, 2018; Holland Michel, Decisions, Decisions, Decisions (note 23); and Human Rights Watch, A Hazard to Human Rights: Autonomous Weapon Systems and Digital Decision-making (Human Rights Watch: New York, Apr. 2025).
27. 27 Milaninia (note 24).
28. 28 Afina, Y., ‘Combat code compliance: International humanitarian law and the development stages of AI for targeting’, Thesis submitted for the degree of PhD in Law (University of Essex, Nov. 2024), p. 87.
29. 29 Blanchard and Bruun (note 20); ICRC, ‘Autonomy, artificial intelligence and robotics: Technical aspects of human control’, 6 June 2019; Holland Michel, Decisions, Decisions, Decisions (note 23); and Bartoletti and Xenidis (note 24).
30. 30 Views expressed during the SIPRI expert workshop (note 22).
31. 31 Mardini, R., ‘Persons with disabilities in armed conflicts: From invisibility to visibility’, International Review of the Red Cross, vol. 105, no. 922 (2023).
32. 32 Blanchard and Bruun (note 20); Bartoletti and Xenidis (note 24); ICRC, ‘Autonomy, artificial intelligence and robotics: Technical aspects of human control’ (note 29); Holland Michel, Decisions, Decisions, Decisions (note 23); and Lai, K. et al., ‘Assessing risks of biases in cognitive decision support systems’, 28th European Signal Processing Conference (EUSIPCO): Proceedings (European Association for Signal Processing: Amsterdam, 2020).
33. 33 Milaninia (note 24); and Holland Michel, Decisions, Decisions, Decisions (note 23).
34. 34 Kwik (note 17), p. 121.
35. 35 Blanchard and Bruun (note 20).
36. 36 AP I (note 14), Arts 41, 48, 51(2), 51(4) and 51(5); and ICRC, CIHL Database (note 15), Rules 1, 6, 7, 13 and 47.
37. 37 AI-DSS have already been reported as having ‘assisted’ users with distinction-related tasks, including identifying, prioritising and nominating targets. See e.g. Nadibaidze, Bode and Zhang (note 1); and Abraham, Y., ‘“Lavender”: The AI machine directing Israel’s bombing spree in Gaza’, +972 Magazine, 3 Apr. 2024.
38. 38 For existing concerns around IHL compliance in the context of AWS see e.g. ICRC, ‘ICRC position on autonomous weapon systems’ (note 9); and CCW GGE on LAWS, ‘State of Palestine’s proposal for the normative and operational framework on autonomous weapon systems’, Working paper submitted by Palestine, CCW/GGE.1/2023/WP.2, 3 Mar. 2023. For existing concerns around IHL compliance in the context of AI-DSS see e.g. Bo, M. and Dorsey, J., ‘The “need” for speed: The cost of unregulated AI decision-support systems to civilians’, Opinio Juris, Apr. 2024; ICRC and Geneva Academy (note 13); and ICRC, ‘Submission to the United Nations Secretary-General on artificial intelligence in the military domain’, 17 Apr. 2025.
39. 39 Shoker, S., Military-age Males in Counterinsurgency and Drone Warfare (Palgrave Macmillan: London, 2021); and Heller, K. J., ‘“One hell of a killing machine”: Signature strikes and international law’, Journal of International Criminal Justice, vol. 11, no. 1 (Mar. 2013).
40. 40 Researcher on international humanitarian law and bias, Interview with the authors, Online, 17 Jan. 2025; and Moyes, R., ‘Target profiles’, Article 36 Discussion Paper, Aug. 2019, p. 7.
41. 41 According to Maya Brehm, ‘constructing profiles on the basis of perceived patterns of behaviour is already fraught with risks [since] the categorization of individuals based on correlations and inferences always entails a certain error rate’, quoted in Moyes (note 40), pp. 6–7.
42. 42 Human Rights Watch (note 26), pp. 44–45 ; and Views expressed during the SIPRI expert workshop (note 22).
43. 43 Dvaladze, G. et al., ‘Beyond the literature, equality and non-discrimination in armed conflict: Humanitarian and human rights law in practice’, International Review of the Red Cross, vol. 926 (Aug. 2024), p. 972; and Human Rights Watch (note 26), pp. 44–45 .
44. 44 SIPRI expert workshop (note 22); and CCW GGE LAWS, ‘Bias in autonomous weapon systems and compliance with international humanitarian law: Addressing a missing link’ Side event, Mar. 2025.
45. 45 Legal expert on IHL and bias, Interview with the authors, Online, 17 Jan. 2025; and Legal expert on IHL and miliary AI, Interview with the authors, Online, 12 Mar. 2025.
46. 46 Moyes (note 40), p. 7. See also Dvaladze, G., Equality and Non-Discrimination in Armed Conflict: Humanitarian and Human Rights Law in Practice (Edward Elgar: Cheltenham, 2023), pp. 235 and 236; Bruun, L., Bo, M. and Goussac, N., Compliance with International Humanitarian Law in the Development and Use of Autonomous Weapon Systems: What Does IHL Permit, Prohibit and Require? (SIPRI: Stockholm, Mar. 2023), p. 11; and ICRC, ‘International humanitarian law and the challenges of contemporary armed conflict’, International Review of the Red Cross, vol. 106, no. 927 (2024) pp. 1452–53.
47. 47 Afina (note 28), pp. 145–47.
48. 48 Views expressed during the SIPRI expert workshop (note 22).
49. 49 Many states and legal experts have expressed the view that to comply with IHL targeting rules, humans must make context-dependent judgements. See e.g. Bruun, L., Towards a Two-tiered Approach to Regulation of Autonomous Weapon Systems: Identifying Pathways and Possible Elements (SIPRI: Stockholm, Aug. 2024), p. 11; CCW GGE on LAWS, ‘Rolling text’, 12 May 2025; ICRC, ‘Submission to the United Nations Secretary-General on artificial intelligence in the military domain’ (note 38); and Lewis, D. A. and Sweeney, H., ‘Exercising cognitive agency: A legal framework concerning natural and artificial intelligence in armed conflict’, Harvard Law School Legal Concept Paper, Jan. 2025.
50. 50 AP I (note 14), Art. 41(1).
51. 51 Sparrow, R., ‘Twenty seconds to comply: Autonomous weapon systems and the recognition of surrender’, International Law Studies, vol. 91 (2015); and ICRC, ‘ICRC position on autonomous weapon systems’ (note 9).
52. 52 Views expressed at the SIPRI expert workshop (note 22).
53. 53 Views expressed at the SIPRI expert workshop (note 22). The view also appears in expert literature; see e.g. Afina (note 28), p. 198.
54. 54 Afina (note 28), p. 226.
55. 55 Horowitz, J., ‘Precautionary measures in urban warfare: A commander’s obligation to obtain information’, ICRC Humanitarian Law & Policy Blog, 10 Jan. 2019.
56. 56 See Longobardo, M., ‘The relevance of the concept of due diligence for international humanitarian law’, Wisconsin International Law Journal, vol. 37, no. 1 (2020), pp. 183–84; Bo, M., Bruun, L. and Boulanin, V., Retaining Human Responsibility in the Development and Use of Autonomous Weapon Systems: On Accountability for Violations of International Humanitarian Law Involving AWS (SIPRI: Stockholm, 2022), p. 9; and Afina (note 28), pp. 224–27.
57. 57 AP I (note 14), Art. 57(2)(a)(i).
58. 58 Klonowska, K., ‘Designing for reasonableness: The algorithmic mediation of reasonableness in targeting decisions’, Articles of War, 23 Feb. 2024); ICRC, ‘Submission to the United Nations Secretary-General on artificial intelligence in the military domain’ (note 38); and Dorsey, J. et al., Academic submission under UN General Assembly Resolution 79/239, at the request of the UN Secretary-General contained in Note Verbale ODA/2025-00029/AIMD.
59. 59 Abraham, ‘“Lavender”: The AI machine directing Israel’s bombing spree in Gaza’ (note 37). See also Dorsey, J. and Bo, M., ‘AI-enabled decision-support systems in the joint targeting cycle: Legal challenges, risks, and the human(e) dimension’, International Law Studies, vol 106 (2025), pp. 24–29; and Abraham, Y., ‘A mass assassination factory”: Inside Israel’s calculated bombing of Gaza’, +972 Magazine, 30 Nov. 2023.
60. 60 Bo, M., ‘Autonomous weapons and the responsibility gap in light of the mens rea of the war crime of attacking civilians in the ICC statute’, Journal of International Criminal Justice, vol. 2, no. 19 (2021), pp. 295–96; ICRC and Geneva Academy (note 13), p. 17; and Dorsey and Bo (note 59).
61. 61 Bo, Bruun and Boulanin (note 56), p. 15.
62. 62 AP I (note 14), Art. 51(5)(b); and ICRC, CIHL Database (note 15), Rule 14.
63. 63 See van den Boogaard, J. (2015), ‘Proportionality and autonomous weapons systems’, Journal of International Humanitarian Legal Studies, vol. 6, no. 2 (Aug. 2015), pp. 261–66.
64. 64 Palantir, ‘Ethical AI in defense decision support systems (Defense AI Ethics, #2)’, Palantir Blog, 27 Nov. 2024; and Palantir, ‘AI, automation, and the ethics of modern warfare’, Palantir Blog, 15 Mar. 2023.
65. 65 Woodcock, T, K., ‘Human/machine(-learning) interactions, human agency and the international humanitarian law proportionality standard’, Global Society, vol. 38, no. 1 (2024); Dorsey, J., ‘Proportionality under pressure: AI-based decision-support systems, the reasonable commander standard and human(e) judgment in targeting’, GC REAIM Expert Policy Note Series, The Hague Centre for Strategic Studies (HCSS), May 2025; and van den Boogaard, J., ‘Proportionality and autonomous weapons systems’, Opinio Juris, 23 Mar. 2016.
66. 66 van den Boogaard, ‘Proportionality and autonomous weapons systems’, Opinio Juris (note 65).
67. 67 ICRC, Gendered Impacts of Armed Conflict and Implications for the Application of International Humanitarian Law (note 4), pp. 7–9.
68. 68 Dvaladze, G. et al. (note 43), p. 957.
69. 69 Dorsey (note 65).
70. 70 Milaninia (note 24), p. 220; and Afina (note 28), p. 85.
71. 71 AP I (note 14), Art. 57(1).
72. 72 Breitegger, A., ‘Towards disability-inclusive IHL: The ICRC’s views and recommendations’, ICRC Humanitarian Law & Policy Blog, 6 July 2023; and Durham, H. et al., ‘Gendered impacts of armed conflict and implications for the application of IHL’, ICRC Humanitarian Law & Policy Blog, 30 June 2022.
73. 73 Bo, Bruun and Boulanin (note 56), p. 15.
74. 74 Common Article 3 of the Geneva Conventions (note 14); GC I, Art. 12; GC II, Art. 12; GC III, Art. 16; GC IV, Arts 13 and 27; AP I, Arts 9, 10, 69, 70 and 75; AP II, Arts 2, 4 and 18; and ICRC, CIHL Database (note 15), ‘Rule 88. ‘Non-discrimination’.
75. 75 Universal Declaration of Human Rights, UN General Assembly Resolution 217 A(III), 10 Dec. 1948, Art. 2.
76. 76 See Dvaladze (note 45), pp. 34, 117 and 123–45.
77. 77 Part III of AP I (note 14), which sets out rules on the means and methods of warfare in international armed conflicts, does not explicitly prohibit adverse distinction in the conduct of hostilities. See also Dvaladze (note 45), pp. 32–45.
78. 78 See Dvaladze (note 45), pp. 45 and 53–55. This position is based on international customary law prohibiting adverse distinction in the application of IHL. See ICRC, CIHL Database (note 15), Rule 88. On state practice see Henckaerts, J. -M. and Beck, L. D., Customary International Humanitarian Law, Volume II: Practice (Cambridge University Press: Cambridge, 2005), pp. 2029–33, referring to several military manuals that recognize the rule on adverse distinction, e.g. French Ministry of Defence, Manual of the Law of Armed Conflict (French Directorate of Legal Affairs: Paris, 2001), pp. 51–52 and Canadian Government, The Law of Armed Conflict at the Operational and Tactical Level (Office of the Judge Advocate General, 1999), p. 2, para. 14.
79. 79 Bode, I. and Bhila, I., ‘The problem of algorithmic bias in AI-based military decision support systems’, ICRC Humanitarian Law & Policy Blog, 3 Sep. 2024.
80. 80 Views expressed during the SIPRI expert workshop (note 22); Kwik (note 17), p. 65; ICRC, ‘Submission to the United Nations Secretary-General on artificial intelligence in the military domain’ (note 38), p. 9; and Afina (note 28), p. 131.
81. 81 Views expressed during the SIPRI expert workshop (note 22).
82. 82 Ferrera (note 18), p. 2; and Just Think, ‘OpenAI Policy Manager: How to mitigate bias in AI’, 21 May 2024.
83. 83 CCW GGE on LAWS, CCW/GGE.1/2024/WP.5 (note 3); Jimenez, A., ‘Gendering the legal review of new means and methods of warfare’, Just Security, 23 Aug. 2022; Dvaladze (note 45), p. 255; Milaninia (note 24); Chandler (note 8); and Asia Pacific Institute for Law and Security, ‘Good Practices in the legal review of weapons, means and methods of warfare’, Version 0.6, Mar. 2025.
84. 84 Views expressed during the SIPRI expert workshop (note 22).
85. 85 See e.g. Haskins, C., ‘The hidden ties between Google and Amazon’s Project Nimbus and Israel’s military’, Wired, 15 July 2024; Palantir, ‘AIP for defense’, [n.d]; and Shanklin, W., ‘OpenAI signs deal with Palmer Luckey’s Anduril to develop military AI’, Engadget, 4 Dec. 2024.
86. 86 Views expressed during the SIPRI expert workshop (note 22).
87. 87 CCW GGE on LAWS, CCW/GGE.1/2024/WP.5 (note 3); and Views expressed during the SIPRI expert workshop (note 22).
88. 88 See Matthews, V. and Murphy, M., ‘Addressing bias in artificial intelligence: The current regulatory landscape’, Thomson Reuters White Paper, 2023, pp. 14–15; and Malik, A., ‘Addressing bias in AI: Strategies for bias mitigation with OpenAI’, Signity Solutions Blog, 22 Apr. 2024.
89. 89 Views expressed during the SIPRI expert workshop (note 22).
90. 90 Views expressed during the SIPRI expert workshop (note 22).
91. 91 Views expressed during the SIPRI expert workshop (note 22). See also, e.g., Just Think (note 82).
92. 92 Views expressed during the SIPRI expert workshop (note 22).
93. 93 Similar rationales are also reflected in Bruun (note 49).
94. 94 Dorsey and Bo (note 59).
95. 95 ICRC, ‘Submission to the United Nations Secretary-General on artificial intelligence in the military domain’ (note 38), p. 11.
96. 96 See ICRC, Reducing Civilian Harm in Urban Warfare: A Handbook for Armed Groups (ICRC: Morges, 2023), pp. 15–16.
97. 97 Views expressed during the SIPRI expert workshop (note 22); CCW GGE on LAWS, ‘ CCW/GGE.1/2024/WP.5 (note 3); and Chandler (note 8).
98. 98 Alexander, P., ‘Exploring bias and accountability in military AI’, LSE Law Review, vol. 7, no. 3 (2022), p. 404.
99. 99 See e.g. Responsible AI in the Military Domain (REAIM) Summit, ‘REAIM Call to Action’, 16 Feb. 2023; CCW GGE on LAWS, CCW/GGE.1/2024/WP.5 (note 3); and Views expressed during the SIPRI expert workshop (note 22).
100. 100 Dorsey and Bo (note 59); and Wood, N., ‘Explainable AI in the military domain’, Ethics and Information Technology, vol. 26, no. 29 (2024).
101. 101 Bartoletti and Xenidis (note 24); and Bode and Bhila (note 79).
102. 102 UK House of Lords, Select Committee on Artificial Intelligence (note 17), p. 44; Bartoletti and Xenidis (note 24); and Howcroft, D., and Rubery, J., ‘“Bias in, bias out”: Gender equality and the future of work debate’, Labour and Industry, vol. 29, no. 2 (2019).
103. 103 Views expressed during the SIPRI expert workshop (note 22); and Chandler (note 8).
104. 104 CCW GGE on LAWS, First session, Seventh meeting (note 5) Intervention by Ireland, 00:45:00.
105. 105 Common Article 1 in the four Geneva Conventions (note 14); and Melzer, N., International Humanitarian Law: A Comprehensive Introduction (ICRC: Geneva, 2022), p. 268.