Contribute

Predictability in the Governance of Autonomous Weapon Systems

Author: Rain Liivoja

Acknowledgements [1]

Introduction

‘Predictability’ features as a key concept in discussions about the regulation of autonomous weapon systems (AWS). Most prominently, the International Committee of the Red Cross (ICRC) has been calling on states to address the risks of AWS by adopting a new legally binding instrument that expressly rules out unpredictable AWS.[2] Many states have supported this approach.[3] Some have put forward alternative proposals that, instead of including unpredictability as an element of a prohibition, have identified predictability as a positive requirement for an acceptable AWS.[4] Multiple other states, however, have not been persuaded.

Consequently, the Group of Governmental Experts on Lethal Autonomous Weapons Systems (GGE) has been struggling to find agreement on language addressing this issue in the ‘set of elements of an instrument’, which the group has been drafting under its 2024–2026 mandate.[5] Early iterations of this document would have required states to ensure that the effects of AWS are ‘adequately predictable, reliable, traceable and explainable’.[6] Having been opposed by several states,[7] this language was dropped from subsequent versions.[8] It then returned as a proposed requirement to ensure that:

[AWS] … in their identification, selection, and engagement functions, are adequately predictable and reliable in operation, and that their functioning and effects are traceable and explainable to those responsible for their development and use.[9]

The rejection of the language referring to predictability by several states does not necessarily mean that they do not see predictability as a desirable attribute of AWS. Rather, they appear to have questions about the precise meaning of this and associated terms. Some states therefore suggest that these terms require further discussion and clarification before being considered for elevation into binding standards.[10] While some of these calls for further terminological debate probably aim to draw out the debate, they are not entirely unfounded. In particular, while multiple national policy documents concerning the responsible use of military AI rely on notions such as reliability, traceability and explainability, they do not use or explain the concept of predictability.[11] Australia’s ‘policy settings’ on military AI are a case in point.[12] Predictability is not one of their ‘values-based principles’ and, indeed, the concept makes no appearance in the document.

This paper considers predictability as a possible normative standard for inclusion in a regulatory instrument on AWS and, by extension, in the governance of military AI. The paper proceeds as follows. After a few notes about vocabulary, it outlines the so-called ‘predictability problem’ with autonomous and AI-enabled systems. Then, it turns to the way predictability has been defined and conceptualised in the literature. Next, it traces the way in which references to predictability have evolved in the GGE (including in relation to existing law) and the emergence of predictability as a possible regulatory standard. Finally, it comments on the advantages and disadvantages of identifying predictability as a discrete standard. The paper concludes that, rather than including predictability as an undefined concept in any new regulatory instrument, it would be preferable to incorporate into such an instrument an articulation of what predictability substantively requires.

Note on Terminology

Before turning to predictability specifically, a brief clarification about the notion of AWS may be helpful, especially as the ‘characterisation’ of AWS has been one of the sticking points throughout the work of the GGE.

This paper adopts a broad understanding wherein AWS are weapon systems that, after activation, can select and engage targets without intervention by a human operator. This broadly aligns with definitions used by the United States and the ICRC.[13]

The characterisation contemplated by the GGE has built upon the same understanding but added additional elements owing to the specific trajectory of debate within the group and given its mandate to deal with ‘lethal’ AWS (often branded with the somewhat unfortunate acronym ‘LAWS’). First, the GGE has introduced target identification as a discrete critical function of an AWS and added a clarification as to its meaning.[14] This paper implicitly follows the ICRC’s approach whereby the system’s ability to select targets presumes and incorporates the capacity to search for, detect, identify and track targets.[15] Second, the GGE’s characterisation also unpacks the notion of ‘weapon system’, which has caused difficulties for some states.[16] Third, the GGE’s characterisation contains a savings clause to ensure that AWS that ‘can be used in a way that does not result in loss of life’ do not fall outside the scope of the instrument.[17] The latter two issues need not be addressed here in detail.

The understanding of AWS adopted in this paper is relatively technology-neutral: it specifies neither the technological means for achieving autonomous functionality nor the precise degree of autonomy in the target selection and engagement functions. On this reading, a landmine is a primitive AWS, and a close-in weapon system (such as Phalanx) is a somewhat more advanced AWS. The GGE has, however, expressly reserved the possibility of excluding certain types of systems from its characterisation, and thus from the scope of its instrument.[18] If this possibility were to be taken up, it would be most likely to affect weapon systems with limited autonomy in their critical functions—for example, automatic contact mines[19] and precision-guided munitions.

In common parlance, AI refers to digital technologies and tools capable of performing tasks normally requiring human intelligence.[20] More formally, AI has been defined as the discipline focused on the research and development of mechanisms and applications of ‘AI systems’, which in turn are engineered systems that generate outputs such as content, forecasts, recommendations or decisions for a given set of human-defined objectives.[21] Currently, machine learning is the most common subset of AI. This approach is focused on algorithms that can ‘learn’ the patterns of training data and, subsequently, make inferences about new data.[22] As a general-purpose technology, AI has a multitude of uses in the military domain across strategic, operational and tactical levels.[23] Among other things, AI can be a powerful ‘enabler’ of autonomy in weapon systems.[24] AI creates ‘improvement opportunities in all application areas of autonomy in weapon systems, from target recognition to navigation’[25] and allows autonomous targeting to ‘leave its former military niche applications and become adoptable across the board’.[26]

Therefore, an AWS may be AI enabled, but it might not be. Conversely, AI may be used to enable some functionality in AWS, but it might not be. That having been said, many of the concerns expressed by states and civil society actors with respect to AWS relate more specifically to AI-enabled AWS, as demonstrated by the pervasiveness of AI-specific concepts in the LAWS GGE discussions.[27] Thus, much of the discussion around the regulation of AWS would also have some bearing on the governance of military AI generally.

The Predictability Problem

Writing in 2004, Andreas Matthias observed that, traditionally, the manufacturer or operator of a machine is held morally and legally responsible for the consequences of its operation.[28] He then identified an ‘accountability gap’:

Autonomous, learning machines, based on neural networks, genetic algorithms and agent architectures, create a new situation, where the manufacturer/operator of the machine is in principle not capable of predicting the future machine behaviour any more, and thus cannot be held morally responsible or liable for it.[29]

A few years later, Rob Sparrow made a similar point regarding AWS:

While [artificially intelligent weapon systems] will be programmed to make decisions according to certain rules, in important circumstances their actions will not be predictable. … The more the system is autonomous then the more it has the capacity to make choices other than those predicted or encouraged by its programmers. At some point then, it will no longer be possible to hold the programmers/designers responsible for outcomes that they could neither control nor predict.[30]

While addressing the ‘accountability gap’, these two papers also offered an articulation of what has now become known as the ‘predictability problem’ (or the ‘unpredictability problem’). At its core, this is a simple but important difficulty: when a system operates without direct real-time human intervention—by sensing its environment, analysing the data, and computing and taking the appropriate course of action—it becomes harder for humans to foresee exactly how it will behave in all circumstances.

Some contemporary authors trace the articulation of the predictability problem back to Norbert Wiener, the founder of cybernetics.[31] The specific point of reference is a 1959 public lecture in which Wiener argued that ‘machines can and do transcend some of the limitations of their designers, and that in doing so they may be both effective and dangerous’.[32] In that lecture, Wiener linked unpredictability to the ability of a system to learn. Specifically, he suggested that ‘[a]s machines learn they may develop unforeseen strategies at rates that baffle their programmers’.[33] For Wiener, it seems to have been the speed and frequency of unforeseen strategies that potentially perplexed humans and became the source of concern, rather than the unforeseeability of individual behaviours more broadly.[34]

Arthur Samuel—the man who coined the term ‘machine learning’—published a refutation of Wiener’s central claims. He posited that ‘[t]he machine is not a threat to mankind, as some people think’, and went on to argue ‘that machines cannot possess originality in the sense implied by Wiener and that they cannot transcend man’s intelligence’.[35] Interestingly, Samuel conceded that neural networks might constitute an exception to this conclusion. He noted that ‘[s]ince the internal connections [of neural networks] would be unknown, the precise behavior of the nets would be unpredictable and, therefore, potentially dangerous’.[36]

Despite the rather longstanding recognition of what appears to be a pivotal problem with AI,[37] in-depth examinations have been rather scarce until recently. Roman Yampolskiy, a pioneer of AI safety research, lamented in 2020 about:

[the] poorly understood concept of unpredictability of intelligent systems, which limits our ability to understand the impact of intelligent systems we are developing and is a challenge for software verification and intelligent system control, as well as AI Safety in general.[38]

He was not alone in identifying a lack of attention to the concept. At roughly the same time, Arthur Holland Michel noted:

[F]or the most part in the discourse on [AWS] and military AI, predictability and understandability have not yet been treated with the kind of detailed foregrounding that befits an issue of such importance and complexity.[39]

The situation has, meanwhile, improved. For example, Yampolskiy himself went on to define the unpredictability of AI as ‘our inability to precisely and consistently predict what specific actions an intelligent system will take to achieve its objectives, even if we know terminal goals of the system’.[40] To give another example, Tobias Vestner and Altea Rossi discussed unpredictability in terms of ‘inherent uncertainty as to how systems will behave once deployed and respond to changing and complex environments’.[41]

Meaning of Predictability

Predictability is, of course, the inverse of unpredictability. It is a common enough word such that, when used in an international law instrument, it should presumptively be read based on its ‘ordinary meaning’.[42] This can be derived from dictionaries. The Oxford English Dictionary defines predictability as ‘[t]he fact or condition of being predictable; the extent to which something can be predicted or used for prediction’.[43] The adjective ‘predictable’ means ‘[a]ble to be predicted or foretold’,[44] and the verb ‘predict’ means ‘[t]o state or estimate, esp[ecially] on the basis of knowledge or reasoning, that (an action, event, etc.) will happen in the future or will be a consequence of something’.[45]

ICRC’s foundational work on the predictability of AWS drew expressly on a similar dictionary definition.[46] On that basis, the ICRC conceptualised the predictability of AWS as ‘knowledge of how the weapon system will function in any given circumstances of use, including the effects that will result’.[47] The reference to ‘knowledge’ creates some challenges. For one, there is a live philosophical debate about whether one can have ‘knowledge’ about what will happen in the future. In more technical terms, this is:

the question whether future contingents are knowable, that is, whether one can know that things will go a certain way even though it is possible that things will not go that way.[48]

Criminal law—where the knowledge of a specific consequence of an act is often an element of an offence—offers something of a way out. Thus, knowledge of a consequence may mean awareness that this consequence will occur ‘in the ordinary course of events’.[49] But this becomes challenging to apply to the functioning of a weapon system generally or with respect to ‘any given circumstances of use’. In any event, the general knowledge of the functioning of a system and the ability to predict its effects under specific conditions seem to be distinct issues.

Mariarosaria Taddeo circumvents some of these problems by simplifying the question: for her, predictability ‘indicates the degree to which one can answer the question: what will an AI system do?’.[50] This understanding is elegant in its simplicity but perhaps slightly vague. Holland Michel characterises predictability as ‘the extent to which a system’s outputs or effects can be anticipated’.[51] This comes reasonably close to the dictionary definition mentioned earlier but, instead of focusing on the ability to ‘estimate’ what will happen in the future, it uses the concept ‘anticipate’, which is more common in the legal vocabulary.[52] Holland Michel’s is probably the single most useful definition, as it clearly acknowledges that predictability (a) relates to anticipating the future, (b) is concerned with effects rather than functions of a system, and (c) is a matter of degree.

The discourse has also sought to identify different types or forms of predictability. Thus, in the ICRC’s view:

There is a … distinction between predictability in a narrow sense of knowing the process by which the system functions and carries out a task, and predictability in a broad sense of knowing the outcome that will result.[53]

With regard to the relationship between these two forms of predictability, the ICRC has taken the view that:

predictability in a broad sense [means] knowing the outcome that will result from activating the autonomous weapon system in a particular circumstance. A sub-component of this is predictability in a narrow sense of knowing the process by which the system functions and carries out a specific task or function.[54]

Again, predictability in the narrow sense here might be better described as the interpretability and explainability of the decision-making process of an AI system. Also, the link between the two seems rather tenuous—a system whose technical functioning is completely opaque might nonetheless be reasonably predictable in the broad sense based on extensive testing and evaluation, and/or operational use.

An alternative approach distinguishes predictability in a technical sense and in an operational sense.[55] From a technical perspective, a system’s predictability refers to ‘the degree of consistency of its past and current behaviour with its future ones’[56]—in other words, the ‘system’s ability to execute a task with the same performance that it [has] exhibited [previously]’.[57] This entails at least two considerations:[58]

  • stability—the extent to which the accuracy of the system’s outputs remains consistent over time
  • generalisability—for machine learning systems, the extent to which the system can handle inputs (data) that deviate from those used for its training and testing.

Technical predictability plainly depends on the system’s architecture and features. For example, systems based on deterministic algorithms, such as decision trees, are more predictable than learning systems,[59] and offline (batch) training models are more predictable than online training models. Put more abstractly, technical predictability depends on the robustness of the system.[60] Robustness here refers to the ability of the system ‘to maintain stable and reliable performance across a broad spectrum of conditions, variations, or challenges, demonstrating resilience and adaptability in the face of uncertainties or unexpected changes’.[61]

From an operational perspective, on the other hand, predictability would refer to ‘the degree to which an autonomous system’s individual actions can be anticipated’[62] or, perhaps more accurately, ‘the degree to which the actions of a system can be anticipated once it is deployed in a specific environment’.[63]

There is broad agreement that all autonomous systems exhibit some degree of inherent operational unpredictability.[64] As a general matter, following Charles Perrow, it is impossible to anticipate all the interactions within tightly coupled complex systems,[65] which autonomous military systems made up of sensors, processors and actuators are likely to be. Moreover, ‘once deployed, systems may face a context with unforeseen characteristics’,[66] leading to interactions with the environment that their designers and operators could not fully anticipate. Indeed, the very point of devising and deploying autonomous systems is that they should be able handle situations of which the operators lack specific prior knowledge.[67]

With respect to AWS, it has therefore been suggested that the functioning and effect of even a deterministic system ‘will be challenging to predict’,[68] and that AWS ‘do not function in a broadly predictable fashion, owing to complexity (in design and task) and interaction with a varying environment’.[69] According to the ICRC:

[A]utonomous weapon systems are unpredictable in a broad sense, because they are triggered by their environment at a time and place unknown to the user who activates them. Moreover, developments in the complexity of software control systems—especially those employing AI and machine learning—may add unpredictability in a narrow sense of the process by which the system functions.[70]

In other words, as noted by Holland Michel:

While technical predictability is solely a function of a system’s performance, operational predictability is just as much a function of the characteristics of the environment and mission for which the system is deployed.[71]

Holland Michel goes on to distinguish a ‘third, general, meaning of un/predictability: the degree to which the outcomes or effects of a system’s use can be anticipated’.[72] This seems to be a distinction without a difference: it is unclear how this ‘third’ form of predictability is any different from operational predictability. The latter appears to precisely capture the ability to anticipate the effects of the system in its operating environment.

Taddeo goes even further. She acknowledges that the predictability problem is ‘multidimensional’ and that the distinction between technical and operational predictability is ‘not really tenable in practice, because technical and operational factors all contribute to determine the behaviour of an AI system’.[73] Indeed, it is the combination of inherent technical factors and extrinsic environmental factors that determines the system’s overall predictability.

Therefore, predictability broadly speaking—capturing both technical and operational dimensions—is influenced by myriad factors such as:[74]

  • the technical features of the system, including the type of algorithm and the level of computing power
  • the complexity of the task or function of the system
  • the complexity and characteristics of the operational environment
  • the robustness of the system, including the quality of its training data and the rigour of testing
  • the system’s capacity to evolve
  • the system’s interaction with other systems
  • the quality of sensor data supplied to the system
  • geographic and temporal scope of the system’s operations
  • the degree to which the operator understands the way the system operates
  • the behaviour of the adversary.

To complicate matters further, ‘[t]hese variables may change and interact together in complex ways, making it difficult to predict all possible outputs of an AI system and their effects’.[75] In short, the degree of overall predictability of a system derives from a complex interaction between the technical capabilities of the system, the nature of its task and the features of its operational environment.

From the Predictability Problem to the Predictability Standard

While acknowledging the early advocacy of technologists and scholars, the governance issues of AWS were propelled onto the multilateral agenda through two major lines of effort, which also supplied the early framing of the problem.

On the one hand, in late 2011, the ICRC articulated publicly its concerns about autonomous functions in weapons. In September that year, Jakob Kellenberger, the ICRC’s president, gave a keynote speech in which he argued that the deployment of AWS ‘would reflect a paradigm shift and a major qualitative change in the conduct of hostilities’.[76] He went on to acknowledge both the challenges in programming such systems to operate consistently with the law and ‘their possible advantages in contributing to greater protection’.[77] The following month, the ICRC included a brief discussion of these issues in its ‘Challenges Report’, released ahead of the 31st International Conference of the Red Cross and Red Crescent, the quadrennial meeting of governments and all components of the Red Cross movement.[78] These early contributions did not, however, expressly mention the predictability problem in relation to AWS.

On the other hand, in April 2013, Christof Heyns, as the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, published an influential report on what he called ‘lethal autonomous robots’ (LARs). [79] In that report, he called, inter alia, for a national moratorium on LARs and for the development of an international governance framework.[80] In his substantive findings, Heyns briefly noted the predictability problem in the following terms:

Autonomous systems can function in an open environment, under unstructured and dynamic circumstances. As such their actions (like those of humans) may ultimately be unpredictable, especially in situations as chaotic as armed conflict, and even more so when they interact with other autonomous systems.[81]

Perhaps the first comprehensive articulation of this problem in relation to AWS appeared in a background paper that the ICRC produced ahead of an expert meeting that it convened in March 2014.[82] Questions about unpredictability and its consequences featured prominently throughout the report of that meeting.[83] Shortly thereafter, in May 2014, the first informal Convention on Certain Conventional Weapons (CCW) Meeting of Experts on Lethal Autonomous Weapons Systems took place.[84] In one of its statements, the ICRC clearly connected the predictability problem to compliance with international humanitarian law (IHL):

If weapon systems become more autonomous—with greater ‘freedom’ to determine their operations—they may become less predictable. Unpredictability would raise serious questions with regard to compliance with IHL. For example: What assurance is there that a weapon system will always operate within the law? How can the weapon system be adequately tested and its performance for its intended use verified without predictability in its likely effects? How can the development and deployment of the weapon system be lawful if there is no guarantee that it will perform in accordance with IHL?[85]

At this stage, however, unpredictability remained predominantly an operational concern rather than a normative concept. Experts were grappling extensively with the preliminary question of characterisation—what kinds of systems were under discussion—rather than with the specific legal consequences of unpredictability. However, several experts and delegations stressed the risks associated with the use of AWS in an operational context, which included the lack of predictability but also vulnerability to cyber attacks and difficulties of adaptation to a complex environment.[86]

The 2015 Meeting of Experts largely followed the same pattern: participants continued to discuss unpredictability as a source of operational and legal risk, albeit on a more technical and granular level. Presentations highlighted the challenges posed by unpredictability to legal reviews and legal compliance generally.[87] Thus, the report of the meeting noted:

Some delegations expressed concern regarding the unpredictability of machine behaviour with self-learning capabilities, in particular when deployed in an unknown or complex environment. The point was made that the lack of determinism and the complexity of the systems would make comprehensive testing difficult, if not impossible. In this context, the importance of limiting the operation of LAWS in time and space was mentioned as a possible solution.[88]

Also, the report noted in cautiously passive voice that the concept of predictability ‘was considered useful for addressing the development of weapon systems, especially those with self-learning capacities’.[89]

In March 2016, the second ICRC-convened expert meeting helped bring further technical understanding and focused in some detail on questions of human control.[90] Notably, in its background paper, the ICRC articulated a link between predictability and human control. It suggested that there are multiple elements that determine whether human control is meaningful, including ‘predictability of the weapon system in its intended or expected circumstances of use’.[91]

The final CCW Meeting of Experts paid significant attention to predictability. Wendell Wallach gave a presentation specifically on that concept, articulating some of the causes of unpredictability and underscoring that predictability is not a binary concept:

Nothing less than a law of physics is absolutely predictable. There are only degrees of predictability, which in theory can be represented as a probability. In the evaluation of weaponry, predictability means that within the task limits for which the system is designed, the anticipated behavior will be realized, yielding the intended result. Nevertheless, an unanticipated event, force, or resistance can alter the behavior of even highly predictable systems. … Member states may differ on the degree of unpredictability and level of risk they will accept in weapon systems.[92]

The increased focus on predictability was duly summarised in the meeting report:

The issue of the predictability of autonomous weapons systems was another important aspect of the debate. It was often framed by the notions of risk, reliability and possible differences between human fallibility and malfunctions of machines. Several delegations expressed concern at the prospect of weapons systems that could act unpredictably. It was argued that the control over a system by a military commander is a core capability for the military and determines the value of such systems. A further point was that the possibility of autonomous ‘swarms’ would mean that such systems would be inherently unpredictable.[93]

Thus, at the time when the informal CCW Meetings of Experts ended and GGE meetings commenced, predictability largely served as a descriptive technical concept rather than a normative standard.

However, in the early work of the GGE, the discussion began to turn to how predictability related to the existing IHL framework. Just to give one example, in a March 2019 statement, the US delegation explained its understanding that the predictability and reliability of AWS are ‘important insofar as they relate to the risk of civilian casualties’ such that an AWS that is ‘unpredictable to the adversary but poses no risk to civilians’ is ‘not a problematic system from the perspective of IHL’.[94] The US delegation also suggested:

[T]he degree of predictability and reliability ‘required by IHL in order to reduce the risk to civilians must be considered in light of all the circumstances, including the operating environment as well as the other precautions that can be taken to reduce the risk to civilians and civilian objects.[95]

Summarising the 2020 meetings of the GGE, the chairperson was able to make the broader point that AWS ‘can pose challenges to compliance with IHL rules since predictability and context-specific human decisions are necessary to ensure compliance, especially bearing in mind current technological limitations’.[96]

A pivotal moment in the discussions arrived in 2021 when the ICRC published its detailed position, referenced at the outset of this article, which expressly called upon states to prohibit unpredictable AWS.[97] This proposal effectively converted (un)predictability from a factor relevant to IHL compliance into a putative freestanding governance standard that could determine a weapon system’s lawfulness. The proposal was plainly influential: predictability subsequently came to feature in proposals for a new CCW protocol submitted by groups of states, as well as, from 2024 onwards, various iterations of the draft elements of an instrument.

Predictability as a Governance Standard

Alignment with Existing Law

The first argument in favour of treating predictability as a governance standard relates to its close alignment with the existing law. Arguably, rules and principles of IHL already imply the need for a weapon system to be predictable, such that articulating predictability as a requirement may be better understood as a clarification of current law rather than as an imposition of an entirely new requirement. This is an important consideration in circumstances where some states resist suggestions that the existing law is somehow insufficient.

Predictability can be grounded in multiple rules currently in force. The prohibition of inherently indiscriminate weapons provides perhaps the strongest anchoring point.[98] This rule prohibits the employment of means and methods of warfare that are of a nature to strike military objectives and civilians or civilian objects without distinction either because they cannot be directed at a specific military objective or because their effects cannot be limited as required by law.[99] Traditionally, this prohibition has been read to preclude the use of means and methods that are highly inaccurate or have uncontrollably propagating effects (such as fire or disease).[100] But the prohibition is agnostic as to the precise cause of the indiscriminate effects.[101] Thus, an operationally unpredictable AWS, where the operator cannot anticipate whether only military objectives would be engaged by the system, satisfies the definition of an inherently indiscriminate weapon and its use is therefore prohibited.

The rules and principles governing the conduct of hostilities also appear to necessitate predictability. Complying with the principle of distinction, the rule of proportionality and the obligation to take precautionary measures[102] requires the operator to make certain legal judgements during or ahead of an attack. To do this, the operator must be able to reasonably anticipate the effects of the weapon in the circumstances prevailing at the time.

Finally, compliance with the obligation to legally review weapons[103] presumes that the weapon’s effects are predictable. The legal review is in essence a determination ahead of time whether the weapon is capable of lawful use. That determination relies on testing and evaluation carried out before the weapon is fielded. If the weapon does not perform consistently over time, such testing and any legal determination based on it would be meaningless.

Capacity of Being Operationalised

The concept of predictability also seems capable of being articulated by reference to technical and operational criteria. Various characterisations of appropriate human–machine interaction—such as ‘meaningful human control’, ‘appropriate human judgement’ or ‘context-appropriate human judgement and control’—all suffer from indeterminacy, partly owing to the complexity of the socio-technical relationship that they seek to capture. Attempts to define meaningful human control as human control that is not nominal have only kicked the definitional can down the road.

Predictability, on the other hand, seems more readily assessable through robustness, along with appropriate design requirements, as well as testing and evaluation standards. Thus, predictability could be unpacked as a requirement to ensure, through testing across a representative range of operational requirements, that a weapon’s performance meets some standard of consistency. Whether a system can be adequately certified as being predictable is at least in principle an empirical question—answerable through technical assessment if appropriate standards are developed.

This is helpful not only from a technical perspective but also from a political one. The notion of appropriate human–machine interaction is subject to considerable disagreement. States disagree as to whether human interaction is a regulatory objective in and of itself, or a means by which some overarching objective (especially compliance with the law) should be achieved. Likewise, states seem to disagree over whether specific technical measures are preconditions for achieving some distinct state of ‘human control’, or whether ‘human control’ is simply the shorthand for the bundle of practical measures that reasonable states should take with a view to achieving compliance with the law. Predictability may carry sufficient technical neutrality to avoid these problems.

Challenges

The notion of predictability comes, however, with its own challenges and limitations. Some of them relate to the very concept while others concern its practical applications.

Open-Endedness

While more concrete than ‘meaningful human control’, ‘appropriate human judgement’ or ‘context-appropriate human judgement and control’, the notion of predictability might still not possess the kind of regulatory precision or technical specificity that would be desirable from a key pillar of a governance framework. The notion immediately raises questions as to what must be predictable, by whom and to what confidence level.

The evolution of the language of the GGE has mitigated part of this problem. Earlier in the debates it was common for participants to refer sweepingly to the predictability (or unpredictability) of the technology or the weapon system. For example, one proposal was to prohibit AWS that ‘operate in a manner that cannot be predicted’ and to take measures to ‘guarantee the weapon system’s … predictability’.[104] One phrasing more recently considered by the GGE would require states, more specifically, to ‘[e]nsure the effects of LAWS are adequately predictable … to those responsible for their use’.[105]

This phrasing would resolve several issues. First, it would specify that it is the effects of the weapon system that need to be predictable, not its operation in some more granular sense. In other words, the focus would be squarely on operational rather than technical predictability. Second, the effect must be predictable to those responsible for the weapon’s use—in other words, the operators who are presumably trained and certified to operate the weapon—and not necessarily to other members of the armed forces. Third, the effects must be adequately predictable, acknowledging that predictability is not a one-off phenomenon but is rather a matter of degree.

But problems remain. How far must the predictability of effects reach? Does it extend to second- and third-tier effects? It does not help to say here, as one might do in a general IHL analysis, that one must consider reasonably foreseeable effects. That would just beg the question. Furthermore, several issues—What level of confidence in predictability is required? How are these levels of confidence articulated? Would AWS with different effects call for different levels of confidence?—would need further examination before predictability could amount to a freestanding standard.

Ex Ante Standards

Considering this uncertainty, the word ‘adequately’ does a lot of heavy lifting in the phrasing mentioned above. But as noted earlier, predictability is at least capable in principle of being expressed in measurable technical and operational criteria. Yet that is not the same as saying that such criteria can be easily developed. Indeed, the opposite appears to be true.

For one thing, there does not seem to be a principled basis for setting particular confidence levels in advance. Any criteria would to a certain degree be arbitrary. Also, the process of developing such criteria would raise challenging questions about the degree to which human behaviour is predictable, and whether any comparison between technical behaviour and human behaviour is appropriate.

Furthermore, any specific criteria are likely to run against the pacing problem—any standard that defines what adequate predictability means for a particular class of AWS may become obsolete rapidly as the technology advances. The behaviour of AWS that is genuinely unpredictable now may become more predictable in the future with new interpretability tools.

Conversely, AWS behaviour deemed adequately predictable today may prove to be inadequate tomorrow with the introduction of more capable and more transparent AI models. For example, during World War II, dropping unguided high-explosive bombs from a high-altitude bomber with the aid of the Norden bombsight was regarded as precision bombing.[106] But this is a far cry from the level of precision that can be achieved with contemporary guided munitions.

Emergent Behaviour and Tactical Unpredictability

A related issue is that in the military context, unlike in the civilian setting, maximum possible predictability of a system is not necessarily desirable. The capacity to surprise an adversary contributes to military effectiveness. Thus, an AWS may be attractive from an operational viewpoint precisely because it behaves in a way that the adversary struggles to anticipate and counteract. As Chris Jenks explains:

There is an advantage when an adversary is not sure how an enemy will conduct operations. Conversely, the more predictable a military force is in combat, the more effective an enemy is in countering. As a result, some amount of emergent behavior (a.k.a. unpredictability) may be desirable.[107]

If the behaviour of the AWS is easily predictable to third parties (such as the adversary), those third parties can learn to manipulate the system inputs in order to produce outputs that they desire. That, in turn, may create additional humanitarian risks or reduce the ability to safely operate the system. Edward Hunter Christie and colleagues explain this predicament thus:

[A] machine learning algorithm whose parameters are fully observable by all parties would be explainable and traceable, and thus be compliant with the principle. However, its behaviour will also be—in principle—predictable for all parties, meaning it would violate security needs and offer opportunities to the adversary to predict its actions and gain an unwanted military advantage. For purposes of military effectiveness, a LAWS should be unpredictable for the adversary, and possibly for the party that operates it, within certain limits. The highest degree of military effectiveness would be achieved if the LAWS could successfully deceive the adversary, while remaining within the bounds of IHL and while also posing no danger to the side that operates it.[108]

In short, for the best outcomes, the AWS should be sufficiently predictable to the operator to ensure legal compliance, but sufficiently unpredictable to the adversary to achieve the desired operational effect and prevent manipulation. According to Jenks, ‘[h]ow much and what kind of emergent behavior [may be desirable] depends on the context in which the system operates but also on a risk calculus’.[109]

This problem could be solved from a governance standpoint by defining predictability as relating to the interaction between the AWS and its operator. This is, in fact, something that the GGE had achieved. The first iteration of its elements of an instrument, from July 2024, stipulated in general terms that ‘[t]o ensure that LAWS can be used in compliance with IHL, their effects must be adequately predictable, reliable, traceable and explainable’.[110] The May 2025 version, however, was far more specific: it required states to ‘[e]nsure [that] the effects of LAWS are adequately predictable, reliable, traceable and explainable to those responsible for their use’.[111] By referring to the persons to whom AWS were supposed to be predictable, the text took a step in the right direction. However, the notion of ‘those responsible for their use’ is arguably too vague. Technical specificity could be introduced by referring to ‘operators’ (especially seeing that the GGE refers to the operator in the definition of AWS). Alternatively, using pre-existing legal language, the focal point of predictability could be ‘those who plan or decide upon’,[112] as they bear the obligations to take precautionary measures in the use of the AWS.

Interconnectedness

A further problem is whether predictability constitutes a sufficiently self-contained standard for governance purposes. There are some indications that this might not be the case.

First, it is illuminating how the ICRC, as the most consistent and sophisticated stakeholder on this point, envisages addressing the predictability problem from a regulatory perspective. Fundamentally, the ICRC argues that ‘unpredictable AWS should be expressly ruled out’.[113] In the ICRC’s view, ‘[t]his could best be achieved with a prohibition on AWS that are designed or used in a manner such that their effects cannot be sufficiently understood, predicted and explained’.[114] This suggests that the predictability problem cannot be addressed by simply insisting on a sufficient level of predictability. Rather, the user’s ability to predict the effects of a weapon operates in tandem with their ability to understand and explain those effects.

Second, the GGE LAWS has previously tended to use predictability together with reliability. The more recent language contemplated by the GGE has referred to predictability in conjunction with not just predictability but also, similarly to the ICRC approach, traceability and explainability. This, then, pushes to the fore questions about the meaning of those additional desirable attributes, and their overlap or interconnectedness.

By Way of a Conclusion

The foregoing discussion does not definitively resolve the question as to whether, on balance, predictability makes for a good governance standard in a proposed instrument on AWS. However, the paper unpacks some of the conceptual complexities and challenges around the notion itself. These should at least call for some caution. In particular, it seems cavalier to rattle off concepts like predictability, reliability, traceability and explainability as prerequisites of legal compliance. First, at that level of abstraction, these notions are firmly grounded in the discussions about the governance of military AI, not all AWS (which might or might not be enabled by AI). Second, this paper’s cataloguing of the various ways in which predictability has been articulated suggests that this concept is not uniformly understood.

With this in mind, a better approach might be to ensure that the text of any legal or policy instrument on AWS reflects the requirement of predictability, even if it does not use this label.[115] For example, a requirement to ‘[e]nsure that the effects of LAWS can be adequately anticipated and controlled by those responsible for their use’[116] might capture much of predictability without introducing a term that, by international law standards, remains novel. Admittedly, some of the problems mentioned do not go away—in particular, ‘adequately’ would do the heavy lifting. But the idea of ‘anticipating’ the effects of weapons is firmly rooted in international law and can be interpreted in light of longstanding state practice.

Furthermore, the uncertainties around predictability probably go some way to explaining why national policy documents do not utilise the concept. Thus, the Australian policy settings on military AI—though too little, too late[117]—wisely avoid its inclusion as a discrete principle. But, for better or for worse, predictability has substantial international traction. Therefore, further work to articulate its interrelationship with nationally endorsed AI governance principles might be helpful for multilateral norm development.

Endnotes

[1] I am grateful to Zena Assaad and Neil Renic for the opportunity to present this paper at a workshop held at the Australian National University, and to Jessica Dorsey, Natalia Jevglevskaja and Lauren Sanders for their insightful and helpful comments on earlier drafts. All views and any errors are mine alone.

[2] International Committee of the Red Cross, ICRC Position on Autonomous Weapon Systems and Background Paper (2021), at: www.icrc.org/en/download/file/166330/icrc_position_on_aws_and_backgroun….

[3] See e.g. Argentina et al., Working Paper, UN Doc. CCW/GGE.1/2022/WP.4 (8 August 2022), para. 11.

[4] See e.g. Costa Rica et al., Joint Working Paper (CCW GGE LAWS, June 2021), p. 3, at: https://documents.unoda.org/wp-content/uploads/2021/06/Costa-Rica-Panama-Peru-the-Philippines-Sierra-Leone-and-Uruguay.pdf; Argentina et al., Draft Protocol on Autonomous Weapon Systems (Draft Protocol VI), UN Doc. CCW/GGE.1/2023/WP.6 (11 May 2023), s. 1.5.

[5] On the mandate, see Meeting of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, Final Report, UN Doc. CCW/MSP/2023/7 (23 November 2023), para. 20.

[6] Cf. Group of Governmental Experts on Lethal Autonomous Weapons Systems, Rolling Text (26 July 2024), box III, para. 6, at: https://perma.cc/T27N-FMUT; GGE on LAWS, Rolling Text (8 November 2024), box III, para. 6.A, at: https://perma.cc/7UM6-TAX2; GGE on LAWS, Rolling Text (12 May 2025), box III, para. 6.A, at: https://perma.cc/Z8Y9-SD9E.

[7] Laura Varella, ‘Report of the Meeting of the Group of Governmental Experts on Lethal Autonomous Weapons Systems’, CCW Report 13, no. 2 (2025): 17, at: https://reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2025/gge/reports/CCWR13.2.pdf.

[8] GGE on LAWS, Rolling Text (18 December 2025), at: https://perma.cc/5G4J-JST4; ‘Additional Suggestions from the Chair for Consideration on 5 March 2026 regarding Boxes I–III of the Rolling Text’ (4 March 2026), at: https://perma.cc/8GP6-T8P3.

[9] GGE on LAWS, Rolling Text (5 June 2026), para. 15, subpara. C.

[10] Varella, ‘Report of the Meeting of the Group of Governmental Experts on Lethal Autonomous Weapons Systems’, p. 17.

[11] For a comparative overview of the principles, see e.g. Rain Liivoja, ‘Principles of Responsible Military Artificial Intelligence and Applicable International Law’, GC REAIM Expert Policy Note Series (Hague Centre for Strategic Studies, May 2025), at: https://hcss.nl/wp-content/uploads/2025/05/Liivoja.pdf.

[12] Department of Defence, Policy Settings for Responsible Use of Artificial Intelligence in Defence: Responsible Use of AI at All Stages of the Technology Lifecycle (Canberra: Commonwealth of Australia, 2025), at: www.defence.gov.au/sites/default/files/2026-03/Policy-Settings-for-Resp….

[13] See e.g. Department of Defense, DoD Directive 3000.09: Autonomy in Weapon Systems (US Government, 2023), at: www.esd.whs.mil/portals/54/documents/dd/issuances/dodd/300009p.pdf; cf. International Committee of the Red Cross, Views of the International Committee of the Red Cross (ICRC) on Autonomous Weapon Systems (Meeting of Experts on LAWS, 2016), p. 1, at: www.icrc.org/en/download/file/21606/ccw-autonomous-weapons-icrc-april-2….

[14] GGE on LAWS, Rolling Text (5 June 2026), para. 1, subpara. A.

[15] International Committee of the Red Cross, Views of the ICRC on Autonomous Weapon Systems, p. 1.

[16] GGE on LAWS, Rolling Text (5 June 2026), para. 1 (‘a combination of one or more weapons and functionally integrated technological components’).

[17] Ibid., para. 1, subpara. B.

[18] Ibid., para. 1, subpara. C.

[19] See e.g. US remarks summarised in Laura Varella, ‘Report on the Virtual Informal Consultation of the Group of Governmental Experts (GGE) on Lethal Autonomous Weapons Systems (LAWS)’, CCW Report 13, no. 3 (2025): 6, at: www.reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2025….

[20] See e.g. BJ Copeland, ‘Artificial Intelligence’, Encyclopaedia Britannica, 20 March 2026, at: www.britannica.com/technology/artificial-intelligence; Office for Artificial Intelligence, National AI Strategy (London: UK Government, 2021), at: www.gov.uk/government/publications/national-ai-strategy.

[21] Information Technology—Artificial Intelligence—Artificial Intelligence Concepts and Terminology, International Standard ISO/IEC 22989:2022(E), July 2022, para. 3.1.3–3.1.4, at: www.iso.org/standard/74296.html.

[22] Dave Bergmann, ‘What Is Machine Learning?’, Think, 18 August 2025, at: www.ibm.com/think/topics/machine-learning.

[23] For an overview, see e.g. Global Commission on Responsible AI in the Military Domain, Responsible by Design: Strategic Guidance Report on the Risks, Opportunities, and Governance of Artificial Intelligence in the Military Domain (HCSS, 2025), pp. 11–19, at: https://hcss.nl/report/gc-reaim-responsible-by-design-strategic-guidance-report.

[24] See e.g. Frank Sauer, ‘Stepping Back from the Brink: Why Multilateral Regulation of Autonomy in Weapons Systems Is Difficult, yet Imperative and Feasible’, International Review of the Red Cross 102, no. 913 (2020): 241, at: https://doi.org/10.1017/S1816383120000466.

[25] Vincent Boulanin and Maaike Verbruggen, Mapping the Development of Autonomy in Weapon Systems (SIPRI, 2017), p. 17, at: www.sipri.org/publications/2017/other-publications/mapping-development-….

[26] Sauer, ‘Stepping Back from the Brink’, p. 241.

[27] See e.g. Ishmael Bhila, ‘Putting Algorithmic Bias on Top of the Agenda in the Discussions on Autonomous Weapons Systems’, Digital War 5, no. 3 (2024): 201–212, at: https://doi.org/10.1057/s42984-024-00094-z.

[28] Andreas Matthias, ‘The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata’, Ethics and Information Technology 6, no. 3 (2004): 175, at: https://doi.org/10.1007/s10676-004-3422-1.

[29] Ibid., p. 175.

[30] Robert Sparrow, ‘Killer Robots’, Journal of Applied Philosophy 24, no. 1 (2007): 65, 70, at: https://doi.org/10.1111/j.1468-5930.2007.00346.x.

[31] Mariarosaria Taddeo and Alexander Blanchard, ‘Accepting Moral Responsibility for the Actions of Autonomous Weapons Systems: A Moral Gambit’, Philosophy and Technology 35, no. 3 (2022): 6, at: https://doi.org/10.1007/s13347-022-00571-x.

[32] Norbert Wiener, ‘Some Moral and Technical Consequences of Automation’, Science 131, no. 3410 (1960): 1355–1358, at: https://doi.org/10.1126/science.131.3410.1355.

[33] Ibid., p. 1355.

[34] I am grateful to Jessica Dorsey for drawing my attention to this point.

[35] Arthur L Samuel, ‘Some Moral and Technical Consequences of Automation: A Refutation’, Science 132, no. 3429 (1960): 741–742, at: https://doi.org/10.1126/science.132.3429.741.

[36] Ibid., p. 741.

[37] Indeed, ‘the predictability problem is central to any analysis of ethical implications of AI in defence’. Mariarosaria Taddeo, The Ethics of Artificial Intelligence in Defence (Oxford University Press, 2024), p. 3, at: https://doi.org/10.1093/oso/9780197745441.001.0001.

[38] Roman V Yampolskiy, ‘Unpredictability of AI: On the Impossibility of Accurately Predicting All Actions of a Smarter Agent’, Journal of Artificial Intelligence and Consciousness 7, no. 1 (2020): 109, at: https://doi.org/10.1142/S2705078520500034.

[39] Arthur Holland Michel, The Black Box, Unlocked: Predictability and Understandability in Military AI (UNIDIR, 2020), p. 1, at: https://unidir.org/publication/the-black-box-unlocked.

[40] Yampolskiy, ‘Unpredictability of AI’, p. 110.

[41] Tobias Vestner and Altea Rossi, ‘Legal Reviews of War Algorithms’, International Law Studies 97, no. 1 (2021): 535, at: https://digital-commons.usnwc.edu/ils/vol97/iss1/26.

[42] See Vienna Convention on the Law of Treaties, 23 May 1969, 1155 UNTS 331, art. 31.

[43] Oxford English Dictionary, ‘predictability (n.)’, last modified July 2023, at: https://doi.org/10.1093/OED/2868482080.

[44] Oxford English Dictionary, ‘predictable (adj.)’, last modified March 2026, at: https://doi.org/10.1093/OED/7210877849.

[45] Oxford English Dictionary, ‘predict (v.)’, last modified December 2025, para. 1.a, at: https://doi.org/10.1093/OED/1009911946.

[46] See e.g., Neil Davison, ‘A Legal Perspective: Autonomous Weapon Systems Under International Humanitarian Law’, in Perspectives on Lethal Autonomous Weapon Systems, UNODA Occasional Paper No. 30 (UNODA, 2017), p. 10, at: www.icrc.org/sites/default/files/document/file_list/autonomous_weapon_s….

[47] International Committee of the Red Cross, Autonomy, Artificial Intelligence and Robotics: Technical Aspects of Human Control (Geneva, 2019), p. 10, at: https://www.icrc.org/en/download/file/102852/autonomy_artificial_intell…; relying on Davison, ‘A Legal Perspective’, p. 10.

[48] Andrea Iacona, ‘Knowledge of Future Contingents’, Philosophical Studies 179, no. 2 (2022): 447, at: https://doi.org/10.1007/s11098-021-01666-5.

[49] Rome Statute of the International Criminal Court, 17 July 1998, 2187 UNTS 90, art. 30(3).

[50] Taddeo and Blanchard, ‘Accepting Moral Responsibility’, p. 6 (italics omitted).

[51] Holland Michel, The Black Box, Unlocked, p. 5.

[52] See e.g. Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of International Armed Conflicts, 8 June 1977, 1125 UNTS 3 (Additional Protocol I), art. 51(5)(b), art. 57(2)(a)(iii) and art. 85(3)(c).

[53] International Committee of the Red Cross, Autonomy, Artificial Intelligence and Robotics, p. 27 (emphases removed).

[54] Ibid., p. 10.

[55] Holland Michel, The Black Box, Unlocked, p. 5.

[56] Taddeo and Blanchard, ‘Accepting Moral Responsibility’, p. 7.

[57] Holland Michel, The Black Box, Unlocked, p. 7.

[58] Ibid., p. 5; Taddeo and Blanchard, ‘Accepting Moral Responsibility’, p. 7.

[59] International Committee of the Red Cross, Autonomy, Artificial Intelligence and Robotics, p. 13; Taddeo and Blanchard, ‘Accepting Moral Responsibility’, p. 7.

[60] Taddeo and Blanchard, ‘Accepting Moral Responsibility’, p. 7.

[61] Houssem Ben Braiek and Foutse Khomh, ‘Machine Learning Robustness: A Primer’, in Marco Lorenzi and Maria A Zuluaga (eds), Trustworthy AI in Medical Imaging (Academic Press, 2025), p. 37, at: https://doi.org/10.1016/B978-0-44-323761-4.00012-2.

[62] Holland Michel, The Black Box, Unlocked, p. 5.

[63] Taddeo, The Ethics of Artificial Intelligence in Defence, p. 5 (emphasis added).

[64] Holland Michel, The Black Box, Unlocked, p. 5.

[65] See generally Charles Perrow, Normal Accidents: Living with High-Risk Technologies, updated edition (Princeton University Press, 2011), at: https://doi.org/10.1515/9781400828494.

[66] Taddeo and Blanchard, ‘Accepting Moral Responsibility’, p. 7.

[67] Holland Michel, The Black Box, Unlocked, pp. 5, 24, n. 8.

[68] Vincent Boulanin, Neil Davison, Netta Goussac and Moa Peldán Carlsson, Limits on Autonomy in Weapon Systems: Identifying Practical Elements of Human Control (SIPRI and ICRC, 2020), p. 7, at: https://www.sipri.org/sites/default/files/2020-06/2006_limits_of_autono….

[69] International Committee of the Red Cross, Autonomy, Artificial Intelligence and Robotics, p. 12.

[70] Ibid., p. 11.

[71] Holland Michel, The Black Box, Unlocked, p. 5.

[72] Ibid., p. 6.

[73] Taddeo, The Ethics of Artificial Intelligence in Defence, p. 6.

[74] Holland Michel, The Black Box, Unlocked, pp. 6–7; Taddeo, The Ethics of Artificial Intelligence in Defence, pp. 5–6.

[75] Taddeo, The Ethics of Artificial Intelligence in Defence, pp. 5–6.

[76] Jakob Kellenberger, ‘Keynote Address—International Humanitarian Law and New Weapon Technologies: 34th Round Table on Current Issues of International Humanitarian Law, San Remo, 8–10 September 2011’, International Review of the Red Cross 94, no. 886 (2012): 812, at: https://doi.org/10.1017/S1816383112000793.

[77] Ibid., pp. 812–813.

[78] International Committee of the Red Cross, International Humanitarian Law and the Challenges of Contemporary Armed Conflicts: 31st International Conference of the Red Cross and Red Crescent, 28 November–1 December 2011, Doc. 31IC/11/5.1.2 (ICRC, 2011), pp. 39–40.

[79] Christof Heyns, Report of the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, UN Doc A/HRC/23/47 (2013).

[80] Ibid., paras 113–126.

[81] Ibid., para. 42.

[82] International Committee of the Red Cross, Autonomous Weapon Systems: Technical, Military, Legal and Humanitarian Aspects (Expert Meeting, Geneva, Switzerland, 26–28 March 2014) (ICRC, 2014), pp. 57–94, at: https://www.icrc.org/en/download/file/1707/4221-002-autonomous-weapons-….

[83] Ibid.

[84] Report of the 2014 Informal Meeting of Experts on Lethal Autonomous Weapons Systems (LAWS), UN Doc CCW/MSP/2014/3 (2014).

[85] International Committee of the Red Cross, ‘Statement of the International Committee of the Red Cross’, Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, 13 May 2014, p. 3, at: https://unoda-documents-library.s3.amazonaws.com/Convention_on_Certain_Conventional_Weapons_-_Informal_Meeting_of_Experts_(2014)/ICRC_MX_LAWS_2014.pdf.

[86] Report of the 2014 Informal Meeting of Experts on LAWS, para. 36.

[87] Report of the 2015 Informal Meeting of Experts on Lethal Autonomous Weapons Systems (LAWS), UN Doc CCW/MSP/2015/3 (2015), para. 51.

[88] Ibid., para. 34.

[89] Ibid., para. 42.

[90] International Committee of the Red Cross, Autonomous Weapon Systems: Implications of Increasing Autonomy in the Critical Functions of Weapons (Expert Meeting, Versoix, 15–16 March 2016) (ICRC, 2016), p. 83.

[91] Ibid.

[92] Wendell Wallach, ‘Predictability and Lethal Autonomous Weapons Systems (LAWS)’, Meeting of Experts on LAWS, Geneva, 12 April 2016, at: https://perma.cc/5JMG-HCR7.

[93] Report of the 2016 Informal Meeting of Experts on Lethal Autonomous Weapons Systems (LAWS), UN Doc CCW/CONF.V/2 (2016), para. 40.

[94] United States of America, ‘Consideration of the Human Element in the Use of Lethal Force’, GGE LAWS, Geneva, 26 March 2019, at: https://geneva.usmission.gov/2019/03/27/convention-on-certain-conventional-weapons-consideration-of-the-human-element-in-the-use-of-lethal-force.

[95] Ibid.

[96] Chairperson’s Summary, UN Doc. CCW/GGE.1/2020/WP.7 (2021), para. 6.

[97] International Committee of the Red Cross, ICRC Position on Autonomous Weapon Systems and Background Paper, p. 2.

[98] See e.g. ibid., p. 7.

[99] See Additional Protocol I, art. 51(4)(b)–(c).

[100] Claude Pilloud and Jean de Preux, ‘Protocol I—Article 51. Protection of the Civilian Population’, in Yves Sandoz et al. (eds), Commentary on the Additional Protocols to the Geneva Conventions (ICRC, 1987), at: https://ihl-databases.icrc.org/en/ihl-treaties/api-1977/article-35/commentary/1987.

[101] Aleksi Kajander, Rain Liivoja and Maarja Naagel, ‘Cybersecurity of Weapon Systems: International Law Requirements and Technical Standards’, Journal of Cybersecurity 11, no. 1 (2025), at: https://doi.org/10.1093/cybsec/tyaf017.

[102] See especially Additional Protocol I art. 48, art. 51(5)(b) and art. 57.

[103] See ibid., art. 36.

[104] Argentina et al., Draft Protocol on Autonomous Weapon Systems (Draft Protocol VI), UN Doc CCW/GGE.1/2023/WP.6 (2023), arts 3, 4(1)(5).

[105] GGE on LAWS, Rolling Text (12 May 2025), box III, para. 6.A.

[106] Raymond Patrick O’Mara, Rise of the War Machines: The Birth of Precision Bombing in World War II (Naval Institute Press, 2022).

[107] Chris Jenks, ‘Responsible AI Symposium—The AI Ethics Principle of Responsibility and LOAC’, Articles of War, 21 December 2022, at: https://lieber.westpoint.edu/ai-ethics-principle-responsibility-loac.

[108] Edward Hunter Christie, Amy Ertan, Laurynas Adomaitis and Matthias Klaus, ‘Regulating Lethal Autonomous Weapon Systems: Exploring the Challenges of Explainability and Traceability’, AI and Ethics 4, no. 2 (2024): 229–245, at: https://doi.org/10.1007/s43681-023-00261-0.

[109] Jenks, ‘Responsible AI Symposium—The AI Ethics Principle of Responsibility and LOAC’.

[110] GGE on LAWS, Rolling Text (26 July 2024), box III, para. 6.

[111] GGE on LAWS, Rolling Text (12 May 2025), box III, para. 6.A (emphasis added).

[112] See Additional Protocol I, art. 57(2)(a).

[113] International Committee of the Red Cross, ICRC Position on Autonomous Weapon Systems and Background Paper, p. 8.

[114] Ibid., p. 8.

[115] Asia-Pacific Institute for Law and Security, ‘Statement on Box III, Paragraph 6’, GGE LAWS, 3 September 2025, at: https://apils.org/2025/09/03/statement-on-box-iii-paragraph-6.

[116] GGE on LAWS, Rolling Text (18 December 2025), box III, para 7(B).

[117] For a more nuanced and eloquent critique, see Netta Goussac and Zena Assaad, ‘Australia’s New Military AI Policy Comes at a Crucial Time. The Challenge Is Turning It into Practice’, The Conversation, 25 March 2026, at: https://doi.org/10.64628/AA.5ctjas7h7.