Contribute

Built In or Bolted On

Industry’s Role in Designing Human–Machine Interaction in AI-Enabled Land Warfare

Author: Lena Trabucco

Introduction: Human Interaction in AI-Enabled Land Warfare

In early 2026, the dispute between Anthropic and the United States Department of Defense (US DoD) revealed a certain fragility within industry–government relationships regarding the development of military artificial intelligence (AI). Put simply, the dispute resulted in Anthropic refusing to authorise the use of its models for two specific use cases—fully autonomous weapon systems and mass domestic surveillance. The basis for the refusal was that these use cases ultimately violated the company’s usage policies and its assessment of what current AI technology can safely and reliably do.[1] The US DoD argued that, from its perspective, it was not appropriate for Anthropic to predetermine the use cases for the Pentagon as long as the models were employed for ‘any lawful use’, and internal policies and regulations would guide US DoD use of, for example, fully autonomous weapons.[2] When Anthropic refused this sweeping access, the department designated Anthropic a supply-chain risk—a label conventionally reserved for foreign adversaries and never previously applied to a US company—and directed federal agencies to cease using its products.[3] However, equally important was the signal to the wider defence-technology industry—grant unrestricted access or face potentially significant consequences.

This episode has important implications for the future of government–industry relationships across the globe. The US DoD signalled its intent to employ AI in contested use cases, or at least to preserve the flexibility to do so, and was willing to impose serious consequences on an American company that declined. Industry, in turn, now has legitimate reasons for caution in partnering with the US DoD or other defence organisations on the development of such systems, particularly the firms best positioned to deliver reliable AI.

Understanding the contours of these relationships matters because industry has an irreplaceable role in developing reliable, responsible and lawful AI systems. Anthropic’s original position rested on two concerns: that frontier AI is not yet reliable enough for fully autonomous weapons to function predictably, marking a red line for the company, and that there is a significant risk that commanders would delegate targeting decisions to machines. As Anthropic CEO Dario Amodei explained, the issue is that there is a need to preserve ‘the right for military officers to make decisions about war themselves and not turn it over to machines completely’.[4] Clearly, one of the core elements of this dispute is the appropriate role for human decision-making and the question of who gets to decide the boundaries of human–machine interaction. And to be sure, the issue and understandings of human control are valid and have been thoroughly analysed elsewhere.[5] However, the unique role of industry in curating a decision space, and the role of human judgement within that space, remains underexplored. This article addresses this gap with a particular focus on land warfare.

AI is already influencing military operations not only through autonomous platforms but also through a growing range of decision-support systems that inform human decision-making.[6] In contemporary and emerging land operations, AI is already, or is likely to be, employed in a range of use cases, such as intelligence analysis, target recognition, sensor fusion, logistics, mission planning, and battlefield decision support, among others.[7] This level of AI integration points to a core concern for armies, which is less a question of autonomy and autonomous weapons, and more a question of how human commanders and soldiers will operationalise human judgement if AI increasingly shapes the information on which operational decisions are based.

For the Australian Army, this question is especially pressing.[8] The Australian Army and the ADF as a whole have repeatedly recognised the importance of addressing human–machine interaction at the outset. Put simply, the human-control issues most pressing in a land environment differ, or at least have different dimensions, from those in maritime or air operations. Land warfare is marked by rapidly changing conditions, degraded communication environments, and greater uncertainty and complexity, particularly in urban environments. The introduction of AI systems —both decision-support systems and weapon systems—carries risks and opportunities for human control in land operations. Those risks and opportunities are explored elsewhere; this article focuses on the role industry plays in addressing the human-control issues specific to land operations.

The Australian Army has been at the forefront of addressing these questions. Numerous doctrinal and policy sources note the importance of incorporating AI while preserving space for human judgement. For example, the ADF Concept for Command and Control of the Future Force recognises that emerging technologies are reshaping the conduct of command and control and that Defence must adapt its concepts, education and capability development accordingly in order to ensure operator awareness and understanding.[9] Additionally, the Army’s Accelerated Warfare: Futures Statement for an Army in Motion frames adaptation to rapid technological and strategic change as a core challenge for the Army, emphasising the need to connect strategy, capability and innovation more effectively.[10] Following these larger perspectives, the Army’s Robotic and Autonomous Systems Strategy calls for robotic and autonomous systems (RAS) to ‘present information in a useable and intuitive way to the soldier’.[11] Finally, the recent policy strategy for responsible use of AI in defence makes it clear that AI is not a niche technical issue but part of a broader transformation in how the Australian Army is commanded, coordinated and employed.[12] All of this guidance is necessary for building the policy and doctrinal foundations necessary for responsible AI integration and curating human–machine interaction, but there undoubtedly needs to be more specific guidance on design parameters for industry to develop Australian military AI. This article takes a step towards defining what that guidance could include.

The Australian Army and partner nations are right to highlight the acceleration of operations, and equally right to ask harder questions about the integration of AI into land operations specifically. To be sure, there are domain-specific considerations regarding the deployment of AI and the role of human judgement. The Army must prioritise a more human-centric focus as it equips and prepares soldiers, whereas other domains, such as air and sea, are more platform-centric. This is not to suggest that there are no important implications for human–machine interaction, but there are notable differences in these approaches.[13] This distinction is also important because Army systems must function in environments where the assumptions underpinning AI models’ development and training are likely at their weakest. Land operations are often conducted under unstable conditions, such as intermittent communications, a contested electromagnetic spectrum, degraded sensor inputs, incomplete data and high risks of adversarial deception.[14] If AI systems are not developed, trained and tested for such conditions, the assumptions that industry has established in system design are unrealistic for these challenging conditions. This would result in AI systems becoming less reliable at critical moments when commanders are under pressure to act. Human interaction, therefore, cannot be understood merely as a matter of preserving the opportunity for a human to decide, but rather as the ability of commanders and operators to employ informed judgement and retain authority when systems are uncertain, partial or wrong. Industry plays a critical role in achieving this.

This article argues that preserving human control in AI-enabled systems depends on designing human–machine interaction so that informed judgement is preserved in contested operational conditions. In this context, the central challenge is not whether the human stays somewhere in the decision chain, or ‘in the loop’, but whether the human can appropriately judge, interpret, challenge and override the system’s recommendations or actions when required. Because much of this capability is designed and delivered outside the Army itself, the defence industry—in Australia and abroad—plays a decisive role in shaping whether human control is built into the system or merely bolted on through the operator’s interaction with it after deployment.

As discussed further below, this article offers five design mechanisms for preserving informed human judgement: interface design, presentation of uncertainty, override architecture, transparency of data inputs and outputs, and fallback states under degraded conditions. These are not peripheral technical details; nor are they considerations to be left to industry alone, because they are the ultimately practical mechanisms through which human judgement is either preserved or eliminated.[15]

The article proceeds in six parts. The first examines how AI is likely to influence decision-making in contemporary and future land operations. The second considers why degraded and contested environments make human–machine interaction especially difficult in Army operations. The third identifies the design mechanisms through which industry implements the preservation of informed human judgement in practice. The fourth proposes three practical design principles for AI-enabled systems intended for land warfare. The fifth examines implications for the Australian Army specifically. And the sixth concludes.

AI and Decision-Making in Contemporary Land Forces

One of the most transformative effects of AI in land warfare is its integration into the processes by which commanders and operators perceive, interpret and act on information. AI-enabled systems are increasingly employed for tasks such as processing intelligence, sensor fusion, route planning, predictive maintenance, autonomous logistics, operational planning, mission simulation, and support for targeting decisions, to name a few.[16] In other words, AI will have a hugely influential role in land operations through shaping how humans make decisions compared to other applications with autonomous systems. This distinction matters because it shifts attention from questions of machine autonomy to the equally important question of how machine outputs enter command processes and shape human judgement.[17]

Experts similarly argue that the Army’s future relationship with machines is likely to evolve from the use of tools to various forms of human–machine teaming, with important implications for trust, command relationships and organisational adaptation.[18] These arguments suggest that future Army operations will not merely add AI onto existing command arrangements but will involve a more fundamental reconfiguration of how information is produced, shared, interpreted and prosecuted.[19]

This has several implications for land operations. First, AI will compress parts of the decision cycle by processing large volumes of data at a pace beyond the capacity of teams of analysts.[20] That will certainly increase the speed of detection, prioritisation and recommendation outputs. Second, AI can aggregate data from multimodal sources and produce a synthetic operational picture that is likely to be difficult to construct in real time.[21] Third, AI may increasingly recommend courses of action, or have other predictive functions, rather than simply presenting information—examples include flagging likely targets, suggesting routes, identifying anomalies, or predicting likely enemy movement patterns. This is an important distinction in identifying what AI offers that is ‘new’ in data analysis and processing. The significance is not simply in the collection and presentation of information but in the incorporation of that information to suggest tangible action and become a contributing element of the decision space. At that point, the issue is no longer one simply of efficiency but of how human judgement is shaped by machine-generated outputs.[22]

For land forces, the significance of this problem is heightened by the distributed and time-sensitive nature of operations. Tactical commanders often act with incomplete information, limited opportunity for deliberation, and uneven communication with operational command.[23] AI-enabled decision support introduced into this environment may provide substantial benefits towards addressing those challenges, but it may simultaneously create new forms of fragility and reliance. For example, an apparently high-confidence recommendation may obscure the degree to which the system relied on incomplete or degraded inputs.[24] In another example, a common operating picture could conceal the system’s uncertainty rather than showcase it.[25] In another case, a prediction model could produce a seemingly reasonable output when, in fact, its assumptions are no longer applicable. In each case, the human is still present, but the quality of control depends on whether the system reveals or hides its own limits.[26]

This suggests that the key control issue in Army AI is not necessarily the allocation of authority between humans and machines but the quality of the human–machine relationship through which decisions are made. A system that produces recommendations in a way that supports situational awareness, reveals uncertainty and requires critical and informed judgement will ultimately strengthen command. A system that presents outputs as authoritative and contestable only to a limited extent will weaken it, even if a human formally approves every decision.[27] Therefore, commanders do not only need to know why a system behaved as it did after an action has occurred; instead, they need to be able to determine whether its recommendation should be considered at all.

This issue becomes even clearer when considering the specific conditions of degraded and contested land warfare. It is one thing for an AI-enabled system to perform effectively when data are abundant, networks are stable, and sensor coverage and output are reliable. It is another for that same system to remain usable when communications are disrupted, data feeds are partial, or adversaries deliberately manipulate the information environment. The next section, therefore, examines the operational conditions that make human control in land warfare uniquely difficult, and why those conditions should be treated as the baseline for industry to design for.

The Reality of Land Operations: Degradation, Uncertainty, Complexity

As mentioned, a significant point of vulnerability for militaries incorporating AI is that systems are often designed, developed, trained and tested by industry, which has its own set of assumptions and interests.[28] But those assumptions do not necessarily reflect the realistic environmental conditions of land warfare. In conditions marked by intermittent communication or disrupted sensor feeds, for example, the problem of human interaction becomes more acute, as the reliability of machine-generated recommendations becomes more uncertain in critical moments with intense time pressure, and when they stand to benefit most from AI support.[29]

Therefore, it is a pressing and immediate challenge for the human decision-makers to understand what the system is doing when the information environment is fragmentary or corrupted. In those instances, human judgement may be weakened or eliminated when a system presents conclusions without making visible the degraded quality of its inputs, the uncertainty of its outputs or the assumptions underpinning its recommendations. In that sense, degraded environments do not merely create technical problems for AI-enabled systems; rather, they expose whether human interaction and judgement have been designed as an operational function or preserved only as a formal decision point.[30]

For AI-enabled systems, the challenges of land environments create at least three interrelated challenges. The first is data degradation. The quality of performance of AI systems depends on large volumes of timely, high-quality input data to generate reliable classifications, predictions or recommendations. In contested environments, however, data may be incomplete, delayed, contradictory or intentionally manipulated. Under these conditions, a system is likely to produce outputs that appear reasonable and actionable, even though the information supporting them has deteriorated significantly, thereby compromising their accuracy and operational utility. What is unclear is whether the operator can perceive that the system’s picture quality has degraded.[31]

The second challenge is uncertainty in interaction. Human factors research has long demonstrated that operators are vulnerable to over-reliance on automated systems, particularly when decisions must be made quickly and when machine outputs carry an appearance of authority.[32] Some experts note the distinction between misuse, disuse and abuse of automation to show that poor human–machine relationships arise not because humans are absent from the decision process but because systems are trusted beyond their appropriate scope or employed in conditions for which they were not designed.[33] In contested land operations, these dynamics carry particular weight. Accelerated pace and tempo compress the time available for deliberation, while an incomplete information picture increases the cognitive pressure to treat machine outputs as a substitute for judgement rather than a support to and enhancement of it. There is a risk that this will result in a gradual shift in the character of human control.

The third challenge concerns machine confidence in complex operational environments. While the previous challenge focuses on how humans relate to AI systems, the opposite is equally important—how AI systems behave when the operating environment becomes unreliable. AI is valued in military applications precisely because it can manage complexity—sorting, prioritising and recommending at machine speed. Yet in combat conditions, that apparent reduction in complexity may itself introduce operational risk. AI systems trained on historical patterns may continue to generate outdated outputs when the environment has shifted in ways those patterns do not capture. For example, a predictive model trained on an adversary’s past behaviour may continue to produce recommendations based on that training even if doctrinal shifts or other vital signals indicate that the adversary is breaking from past patterns. This introduces new risks for commanders when introducing AI systems and highlights the importance of recognising the alignment of AI systems with the environment in which they are deployed.

Industry has a critical role in addressing these problems. If AI systems do not embed human control mechanisms for degraded environments, human control will be strongest when it is least needed and weakest when it is most needed. By contrast, if systems are designed on the assumptions that communications will be intermittent, data will be partial and adversaries will actively manipulate the information environment, human–machine interaction can be made to support—rather than substitute for—judgement under precisely those conditions.

For the Australian Army, this point has direct capability implications. Australia is unlikely to deploy AI-enabled land systems only in benign or highly connected environments. Any serious consideration of future Army capability must assume operations over wide distances, reliance on coalition networks, and exposure to electronic warfare, cyber disruption, and degraded battlespace awareness. Under those conditions, the value of AI will depend not only on the sophistication of the model but also on whether the system is designed to communicate uncertainty, reveal degraded inputs and inform human judgement when the technical and operational picture is incomplete. Human control, in other words, is tested not when systems perform well in ideal conditions, but when they continue to function effectively even as the basis for trusting them is unclear.

Industry Mechanisms to Preserve Human–Machine Interaction

Industry is critical to tackling these challenges, and if human–machine interaction in AI-enabled land warfare is to be genuine rather than symbolic, it must be deliberately built into the system through thoughtful design. In practice, many of the systems through which the Army will encounter AI are developed not by military organisations themselves but by defence companies, software providers or other research partners. As a result, the practical operationalisation of human interaction is often determined before the system is fielded, with industry ultimately deciding the parameters of system architecture, the design of the interface, the way data are presented, the conditions under which users can intervene, and the mechanisms by which the system fails or reverts under degraded conditions. Therefore, questions of human interaction are a matter not only for commanders and operators but also for engineers and designers, and industry is central to resolving them.

As discussed above, the degraded conditions that characterise land operations pose significant challenges for AI system performance. Ultimately, systems developed for the Army are likely to operate precisely in those conditions where AI reliability is under the greatest pressure. Industry therefore plays a critical role not only by supplying AI-enabled capability but also by embedding specific mechanisms that preserve human judgement in such operationally realistic conditions. This article addresses five mechanisms in particular: interface design, presentation of uncertainty, fallback modes, override architecture, and transparency of data inputs and outputs. The discussion of each mechanism also includes a brief hypothetical example to illustrate the design features in concrete terms.

The first mechanism is interface design. Human control depends heavily on how the system structures and bounds the operator’s decision space. The interface is a critical tool because it does not merely display information—it shapes what the operator notices, what appears important, which courses of action seem available and how much cognitive effort is required to assess an AI recommendation. Consider a concrete example: a route-planning tool used by a tactical commander under significant time pressure. If the interface presents a single AI-generated route as the recommended option, the commander is effectively being asked to approve a conclusion rather than make an informed decision. She may have no visibility into whether the route was generated from current or outdated data, whether it includes the latest reported enemy positions, or how the system weighted competing factors such as speed, cover and network connectivity. While the recommendation appears sound, the amounts of time and cognitive effort required to question it are high. This is not to suggest that the opposite is better—human factors research shows that the solution is not to display more information. Overloading the interface with information such as system assumptions, confidence scores and data provenance risks replacing one human factors problem with another—overwhelming commanders with details when clarity is the priority. Therefore, the design challenge for industry is one of calibration—presenting information in a way that supports informed judgement without sacrificing the speed and cognitive efficiency that make AI decision support valuable in the first place. Getting that calibration right for combat environments requires deliberate, operationally informed design choices for realistic conditions.

The second mechanism is the presentation of uncertainty. One of the greatest risks in AI-enabled military systems is that outputs appear authoritative even when the information supporting them has degraded. In land warfare, this risk is acute because operators work under time pressure and have limited capacity to validate machine-generated recommendations against other independent sources. To be clear, the problem is not that AI systems produce uncertain outputs, because uncertainty is a guaranteed feature in a complex operational environment. The problem is that uncertainty is invisible to the operator who receives and interprets those outputs. For example, consider a targeting recommendation generated by a sensor-fusion system operating in a complex environment. In this case, the operator is unaware that one of the system’s data feeds has dropped entirely and been replaced by stored data from several hours earlier, and that a second sensor has been intermittent for the past 40 minutes. The recommendation the system produces looks the same as it would if all feeds were live and in real time, with the same format and apparent certainty. The operator may have no way of knowing that the picture underlying the recommendation has deteriorated, and—under time-critical pressure to act—she is unlikely to pause and investigate. This example points to an important risk: not of the system being wrong but of the operator having no reliable means of knowing when to be sceptical. The solution is not necessarily to attach a raw statistical confidence score to every output, as research shows this may not provide more clarity to the operator in the field. Industry is better served by designing systems so that uncertainty is communicated in forms that are operationally meaningful. It may seem counterintuitive, but a system will be more helpful if it displays uncertainty and clearly preserves the operator’s capacity to calibrate trust, apply judgement, and intervene when confidence is no longer warranted.

The third mechanism is the design of fallback states.[34] As discussed, AI-enabled systems will not always have access to stable communications, complete data or uninterrupted connectivity. Fallback modes are the conditions a system reverts to when its primary capabilities are no longer available—a degraded state that may offer less analytical capacity or fewer options but keeps the system functional and, importantly, keeps the operator informed about what has changed.[35] A system that continues to produce outputs without telling the operator that its underlying picture has deteriorated may leave the operator in a worse position than if the system had never been introduced at all, because she is now operating without the situational awareness she would have developed through other means. Consider another example. An AI-enabled targeting recommendation system is operating in an urban environment. Over the course of an operation, the system has built a detailed operational picture of a city—identifying new construction, structural changes to buildings, and other features of the urban landscape relevant to targeting decisions. However, the system experiences a malfunction and reverts to its fallback state, losing the operational picture it has collected and defaulting to its baseline map data. That baseline reflects the city as it was when the system was originally configured, before it learned anything during the operation, including about new construction and other structural changes. A poorly designed fallback makes none of this visible to the operator. The system would continue to generate targeting recommendations, and the commander would have no indication that the picture underlying those recommendations no longer reflected the city’s current layout. Strike recommendations drawn from that degraded picture carry both operational and legal risks—such as hitting structures whose function or occupancy has changed, or missing features of the urban environment that have since become operationally significant. A commander cannot assess those risks if she does not know the picture has changed. A well-designed fallback, by contrast, alerts the operator immediately to the malfunction, clearly identifies which data has been lost, and flags that recommendations are now being generated from baseline data that does not reflect the real-time operational picture the system had previously accumulated.

The fourth mechanism is the design of the override architecture. It is common to claim that human control is preserved because a system includes a system override mechanism, especially in ‘human-in-the-loop’ models. However, in practice, having an override mechanism and having access to the information needed to determine its appropriate use are two different things. The mere existence of the ‘stop’ button does not, in fact, preserve human control. The more fundamental design question is not whether an operator can physically activate an override but whether she has sufficient information to know when to do so. Consider a targeting-support system that produces recommendations a commander finds tactically suspect. If the override mechanism operates in isolation from the system’s other data sources, that is, if activating it simply stops the current recommendation without telling the commander why the system arrived at that recommendation, what data it was drawing on or how confident it was in its output, the commander is being asked to exercise judgement without the necessary information. She may or may not override correctly, but either way, the decision is less informed than it should be. A well-designed override architecture needs to be closely integrated with the system’s other transparency functions. For example, when an operator considers intervening, the system might include the information most relevant to that decision, such as what the recommendation was based on, where the data inputs are strong or weak, and, perhaps most significantly, what the operational consequences of an override are likely to be.

The fifth mechanism is transparency of data inputs and outputs. Transparency here means designing the system so that the operator has sufficient context around where a recommendation came from and what its limitations are, enabling an informed decision about how much weight to place on it. There are two dimensions of input/output transparency that are most critical here. The first concerns the data used to train and test the system. A system learns what it is taught, and if the conditions reflected in its training data do not match those in which it is deployed, its recommendations may be unreliable in ways that are not immediately apparent to the operator. For example, imagine a target-recognition system trained predominantly on land-based environments with disproportionately sunny conditions. When that system is deployed on a morning of dense fog, it is being asked to assess and recommend targets in conditions its training data did not adequately represent. If the operator has no way of knowing that the system has limited exposure to low-visibility environments, she cannot account for that limitation in her own judgement—and a recommendation that would warrant scrutiny in fog is accepted with the same confidence as one generated on a clear day. The second dimension concerns the data it is using in real time. In a separate example, imagine a system that estimates civilian presence in an urban area by collecting and analysing mobile phone signals. In a busy marketplace during a local holiday, the volume of signals may exceed the system’s capacity to fully process them, leading the system to underestimate the number of civilians present and to produce targeting information that does not reflect the actual civilian risk on the ground. If the operator is not told that the system’s data collection was incomplete and that the picture of civilian presence it is working from is only partial, then she has no basis for treating that recommendation with the caution it warrants. This is not to suggest that a soldier in the field needs to understand how a model was built. She does need to know, in plain terms, whether the system was designed for the environment she is operating in and whether the picture it is working from reflects the situation as it currently stands.

Taken together, these mechanisms illustrate the critical role of industry in curating human–machine interaction and maintaining informed judgement. Interface design shapes which information is used and how decisions are framed; the presentation of uncertainty shapes how much confidence the operator should have; fallback states shape how systems remain usable and appropriate under degradation; override architecture shapes the decision space for terminating the system appropriately; and transparency of data inputs and outputs shapes whether operators can understand the basis and limits of machine recommendations. Through these mechanisms, industry determines whether informed judgement is embedded in the capability itself or simply understood as the requirement for a human to ‘press a button’. But, as is made clear, it is not this simple.

Designing for Human Control: Three Design Principles for Land-Based AI Systems

The previous section illustrates the critical role of industry in curating human–machine interaction through particular design mechanisms. But industry is currently left to its own devices to make many of the design choices that will ultimately impact the manner of human–machine interaction and how well informed human judgement will be in practice. These outcomes will not simply emerge automatically through technological sophistication or cutting-edge innovation. Whether they occur depends on whether governments, defence organisations, industry and other stakeholders align their efforts early enough to shape systems to remain appropriate and reliable under operational stress. Building on the mechanisms identified in the previous section, this section draws out three principles that should guide industry development. The first is to provide clear top-down guidance early enough to shape design. The second is to re-evaluate industry assumptions in development. The third is to design data presentation to surface limitations.

Provide Clear Top-Down Guidance Early Enough to Shape Design

The first principle is that governments and defence organisations must provide earlier and clearer guidance regarding human–AI interaction in AI-enabled military systems, for both decision-support tools and weapon systems. Without such guidance, industry is left to interpret vague political language or fragmentary policy signals on its own. In that absence, companies will design against whatever procurement requirements already exist—which are likely to be outdated, inapplicable to AI-enabled decision-support systems or silent on questions of human control altogether. A particular risk is that without such guidance, companies will interpret human–machine interaction differently and the range of interpretations will be wide, with varying levels of usefulness. The systems that the Army acquires could reflect that variation, and the practical meaning of human interaction and the impact on human judgement will be shaped less by deliberate policy than by the accumulated design choices of developers working without adequate direction.[36]

To be sure, top-down guidance does not need to constrain industry creativity. Rather, this approach allows industry to channel it appropriately. Clear guidance on expectations or operational requirements for human interaction does not narrow the design space in ways that would stifle innovation. On the contrary, it gives developers a well-defined problem to solve, and difficult, well-defined problems are exactly the conditions under which engineering creativity tends to produce its most useful results. The five design mechanisms offer many such concrete, well-defined problems: how to communicate uncertainty in operationally meaningful terms, how to design fallback states that preserve operator awareness, how to build override architecture that works under dynamic battlefield conditions. These are genuinely hard challenges, and there is no single correct answer. Top-down guidance that frames human interaction as a concrete design requirement invites industry to utilise their expertise to solve these problems, rather than leaving human interaction as an afterthought addressed through compliance language at the end of a procurement process.[37]

If defence organisations want industry to build systems that prioritise human interaction and support informed human judgement, they must signal that expectation early enough to influence design, architecture, interface and testing decisions. Existing policy instruments in the US and UK illustrate both the potential for defence organisations to take this approach and the current limits on how they can do so. For example, the 2023 US DoD Directive 3000.09 establishes senior-level review requirements for autonomous and semi-autonomous weapons but stops short of providing industry-level guidance to help operationalise ‘appropriate levels of human judgement’.[38] Similarly, the UK Ministry of Defence’s 2022 ‘Ambitious, Safe, Responsible’ policy sets out AI ethics principles but leaves implementation largely to developers.[39] For the Australian Department of Defence, the absence of equivalent guidance on AI-enabled system design represents an opportunity rather than simply a gap. The ADF has already recognised this challenge. The Army RAS Strategy calls for the Army to ‘understand the design parameters of AI technology and algorithms [that may be] embedded in the fielded capability [and] may require Australian-derived verifiable-by-design AI, rather than a “black-box” solution’.[40] Offering this top-down guidance would give Australia the chance to put many policies into practice and establish a foundation for the Australian defence industry to develop these capabilities aligned with Australia’s prioritisation of informed human decision-makers.

Re-Examine Industry Assumptions in Design and Development

The second principle is that industry must re-examine the assumptions underpinning the design, development, training and testing of AI-enabled systems intended for land operations. As discussed above, these systems are typically designed, developed, trained and tested by industry actors whose assumptions do not necessarily reflect the realistic environmental conditions of land warfare. For the Army, the relevant operational environment is not one marked by stable and reliable conditions. Hence, systems designed against assumptions suited to permissive or idealised conditions may perform well in demonstrations but prove unsuitable for real combat environments.[41]

Degradation is the clearest example of this problem, but it is not the only one. A system built on the assumption of stable connectivity and reliable sensor coverage will struggle with disrupted communications or a contested information environment. The same failure mode occurs in any context when a system’s training or design assumptions are no longer valid. In such cases, the system does not necessarily fail outright, as it continues to produce outputs, but those outputs rest on assumptions that no longer reflect the operational environment. Re-examining assumptions, therefore, changes the standard against which capability should be judged. This requires industry to consider not only whether the system performs well under the conditions for which it was designed and trained but also whether its output remains appropriate and intelligible when those conditions no longer hold. A system that performs well against its own design assumptions but becomes overconfident or difficult to interpret once those assumptions break down does not preserve human judgement and instead poses substantial risks to the forces that use it. Therefore, industry’s assumptions should be treated not as fixed premises to be addressed only if they fail in the field, but as standing questions to be revisited against the conditions the system will encounter. For the Australian Army, which must plan for operations across wide distances, degraded communications, electronic warfare and coalition network dependencies, few of the assumptions that hold in controlled simulations can be taken for granted in practice.

Design Data Presentation to Surface Limitations

The third principle is that data presentation should be designed to disclose the limitations of the information underlying a system’s output, not simply of the output itself. When such systems are introduced, the significant risk is not necessarily machine error or failure; it is misplaced or unwarranted confidence. A system that produces authoritative-looking outputs while concealing the fragility of the information underpinning them necessarily undermines the operator’s capacity for informed judgement. Designing to surface limitations means that systems should show the operator not only what the machine recommends but where that recommendation came from, how confident the system is, and why it has that level of confidence, in terms the operator can act on. This is essential in land warfare, where decisions are often made quickly with incomplete information, and the consequences of acting on a flawed picture can be severe.

How a limitation is communicated matters as much as whether it is communicated at all. Take, for example, the issue of uncertainty. A raw statistical confidence score—a percentage figure attached to an output—may not carry much operational meaning for an operator, as a percentage may not have the same weight for each individual.[42] It may be more useful to use visual representations of uncertainty that are immediately intelligible in operational terms—ones that convey, at a glance, the basis and quality of the recommendation rather than its statistical properties. The design of that representation is one of the most consequential choices industry makes in determining whether human judgement is genuinely informed.[43]

This principle also has a direct legal dimension. The obligations that international humanitarian law (IHL) places on commanders—to distinguish targets, to assess proportionality and to take feasible precautions before an attack—all depend on the commander’s having an accurate understanding of available information.[44] A system that presents a targeting recommendation with apparent precision while concealing that its underlying picture is based on incomplete, degraded or outdated data actively impedes the commander’s ability to fulfil her legal obligations. Surfacing the system’s limitations is therefore not simply a design preference or a human factors consideration; it is an important component of IHL compliance and operational effectiveness. A system that is technically capable of surfacing limitations but is not designed to do so intelligibly will not result in informed human judgement.

Each of these principles is deliberately broader than any technical feature set. They are intended to guide how Defence proactively communicates the kinds of systems it needs, how industry operationalises those needs, and how Army assesses whether AI-enabled systems genuinely support human judgement in land operations. Ultimately, human interaction cannot be preserved simply by keeping a human in the loop; it must be deliberately designed into the institutional setting, development assumptions and operational behaviour of the system itself. For the Australian Army, that is the more demanding standard but also the more effective one.

Implications for the Australian Army

The foregoing analysis carries three implications for the Australian Army and, more broadly, for the Australian Department of Defence as it implements its Policy Settings for Responsible Use of Artificial Intelligence in Defence.[45] Released in March 2026, the policy settings are Defence’s first dedicated AI governance framework applying across the entire ADF and the wider Australian defence portfolio. They set out Defence’s obligations for responsible AI use under three headings—lawfulness, adherence to values-based principles, and proportionate controls—spanning the full technology life cycle from research and design through to decommissioning. Each policy principle maps nicely to one of the design principles set out above and speaks to a different gap between Defence’s stated commitments and the practical mechanisms through which those commitments must be realised in the systems Army acquires and operates.

The first implication concerns top-down guidance. The three principles from Defence’s policy settings currently operate at a level of generality that leaves industry to interpret what they require in practice. These principles reflect the broader public debate about military AI, which continues to treat autonomous action—particularly autonomous weapon systems—as the principal threat to human control. However, for land forces, the more pressing risk is the inability to promote informed judgement with decision-support systems that never act autonomously, and generalised principles are poorly suited to identifying that risk before systems are fielded. Articulating requirements earlier—specifying not only that human interaction must be preserved but also what that means for practical design features such as interface design, fallback states and override architecture—would translate principle into specification and position Australia as a contributor to an emerging international conversation on responsible AI design and development.

The second implication concerns coalition operations. The Anthropic–DoD dispute offers a clear warning about how unevenly perspectives on human interaction may translate into standards that are likely to be distributed across allies and partners. When a major coalition partner is prepared to designate a domestic AI provider a supply-chain risk for declining to authorise certain use cases, the implications for interoperability could be significant. Coalition operations may increasingly involve systems built on different control assumptions, with different fallback behaviours, different override architectures, and different conventions for surfacing uncertainty or other limitations. For an Australian commander employing coalition decision-support tools, the question is not whether the partner system is simply reliable, but whether its design assumptions are intelligible and compatible with Australia’s own legal and policy commitments.

The third implication concerns data presentation, which is closely linked to the principle of explainability set out in the Policy Settings for Responsible Use of Artificial Intelligence in Defence. That principle requires that the function and the relationship between inputs and outputs be traceable, and that systems meaningfully accommodate human–machine interaction. However, as this article has shown, this can only be achieved if industry designs traceability into the system from the outset rather than treating it as something to be demonstrated after the fact. Australia has already established the policy foundation necessary to move into the next iteration of guidance and to tackle design-level challenges, ensuring that Australian Army commanders and operators have the most reliable systems available for operations.

Conclusions

AI is often discussed in military terms as a question of autonomy: how much independence a system should have, whether humans remain in or on the loop, and where formal responsibility should lie when machines engage in autonomous behaviour. While these questions are undoubtedly significant, a more immediate challenge is how AI-enabled decision-support systems will preserve and inform human judgement under the degraded, contested and time-pressured conditions that characterise land warfare. As this article has argued, human interaction in that context is less about human presence than about whether the human–machine relationship has been deliberately designed to support judgement, understanding, and intervention when necessary.

The article advances three linked claims. First, industry has a critical role in preserving informed human judgement. Second, human–machine interaction is a critical mechanism of control, and mechanisms such as interface design, the presentation of uncertainty, fallback states, override architecture, and the transparency of inputs and outputs determine whether commanders can judge, question and redirect system behaviour in practice. Third, three design principles can guide government and industry in strengthening these capabilities. The Australian defence industry plays a central role, as these mechanisms are determined upstream through capability design rather than downstream through doctrine alone. The practical meaning of human control is therefore shaped not only by command protocols but also by the design decisions built into the systems the Army acquires.

If Australian Defence treats human interaction as a design requirement from the outset, it can shape a different model of military AI that supports informed command rather than displacing it, and in which human judgement remains strongest when operational conditions are at their worst. That is a more demanding approach to capability development, but it is also the one most consistent with the realities of land warfare, responsible AI and the needs of future Australian Army operations.

Endnotes

[1] ‘Usage Policy’, Anthropic (website), at: www.anthropic.com/legal/aup (accessed 15 June 2026).

[2] United States Department of Defense, Autonomy in Weapon Systems, DoD Directive 3000.09 (DoD, 2023).

[3] Hasan Ali Kanu, ‘Hegseth Doubles Down on Anthropic’s Security Risk Designation’, Politico, 4 June 2026, at: www.politico.com/news/2026/06/04/hegseth-anthropic-designation-supply-chain-risk-00951183.

[4] Ibid.

[5] See Lena Trabucco, ‘AI-Enabled Autonomous Weapons and Human Control: Part I: Human Control and Machine Learning Design and Development’, International Law Studies 106 (2025): 533–578, at: https://digital-commons.usnwc.edu/ils/vol106/iss1/17; Anna-Katharina Ferl, ‘Imagining Meaningful Human Control: Autonomous Weapons and the (De-)Legitimisation of Future Warfare’, Global Society 38, no. 1 (2024): 139–155, at: https://doi.org/10.1080/13600826.2023.2233004; Filippo Santoni de Sio and Jeroen van den Hoven, ‘Meaningful Human Control over Autonomous Systems: A Philosophical Account’, Frontiers in Robotics and AI 5 (2018), at: https://doi.org/10.3389/frobt.2018.00015; Heather Roff and Richard Moyes, ‘Meaningful Human Control, Artificial Intelligence and Autonomous Weapons’, briefing paper prepared for the Informal Meeting of Experts on Lethal Autonomous Weapons Systems, UN Convention on Certain Conventional Weapons (2016), at: https://www.stopkillerrobots.org/wp-content/uploads/2021/09/Roff_Moyes_Meaningful_Human_Control-with-cover-page-v2.pdf; Michael Horowitz and Paul Scharre, Meaningful Human Control in Weapon Systems: A Primer (Center for a New American Security, 2015), at: www.cnas.org/publications/reports/meaningful-human-control-in-weapon-systems-a-primer; ‘Killer Robots: UK Government Policy on Fully Autonomous Weapons’, Article 36, 19 April 2013, at: https://article36.org/statements-updates/killer-robots-uk-government-policy-on-fully-autonomous-weapons-2.

[6] Anna Nadibaidze, Ingvild Bode and Qiaochu Zhang, AI in Military Decision Support Systems: A Review of Developments and Debates (Center for War Studies, University of Southern Denmark, 2024), at: www.autonorms.eu/wp-content/uploads/2024/11/AI-DSS-report-WEB.pdf; Paul Scharre, Four Battlegrounds: Power in the Age of Artificial Intelligence (New York: W.W. Norton, 2023); James Johnson, ‘Artificial Intelligence and Future Warfare: Implications for International Security’, Defense and Security Analysis 35, no. 2 (2019): 147–169, at: https://doi.org/10.1080/14751798.2019.1600800.

[7] Australian Army, Robotic and Autonomous Systems Strategy v2.0 (Canberra: Commonwealth of Australia, 2022), at: https://researchcentre.army.gov.au/rico/robotic-and-autonomous-systems/robotic-autonomous-systems-ras-strategy; Alex Neads, Theo Farrell and David J Galbreath, ‘Evolving Towards Military Innovation: AI and the Australian Army’, Journal of Strategic Studies 47, no. 5 (2024): 669–698, at: https://www.tandfonline.com/doi/full/10.1080/01402390.2023.2200588.

[8] Australian Defence Force, ADF Concept for Command and Control of the Future Force, version 1.0 (Canberra: Commonwealth of Australia, 2018), at: https://theforge.defence.gov.au/sites/default/files/adf_concept_for_command_and_control_of_the_future_force_v.1_signed.pdf; Australian Defence Force, Concept for Robotic and Autonomous Systems (Canberra: Commonwealth of Australia, 2020), at: https://defense.info/wp-content/uploads/2021/06/ADF-Concept-Robotics.pdf; Benjamin Wood, ‘Operational AI Integration and Governance in the Australian Army’, Australian Army Journal 21, no. 1 (2025): 117–147, at: https://doi.org/10.61451/1235807.

[9] Australian Defence Force, ADF Concept for Command and Control of the Future Force.

[10] Australian Army, Accelerated Warfare: Futures Statement for an Army in Motion (Canberra: Commonwealth of Australia, 2018), at: https://researchcentre.army.gov.au/library/other/accelerated-warfare.

[11] Australian Army, Robotic and Autonomous Systems Strategy.

[12] Department of Defence, Policy Settings for Responsible Use of Artificial Intelligence in Defence (Canberra: Commonwealth of Australia, 2025), at: https://defencescienceinstitute.com/news/dept-of-defence-policy-settings-for-responsible-use-of-ai-in-defence.

[13] The author thanks a reviewer for bringing this point to her attention.

[14] Mick Ryan, Human-Machine Teaming for Future Ground Forces (Washington DC: Center for Strategic and Budgetary Assessments, 2018), at: https://csbaonline.org/resource/human-machine-teaming-for-future-ground-forces; Jack Watling and Noah Sylvia, Competitive Electronic Warfare in Modern Land Operations (London: Royal United Services Institute, 2023), at: www.rusi.org/explore-our-research/publications/occasional-papers/competitive-electronic-warfare-modern-land-operations.

[15] See also Trabucco, ‘AI-Enabled Autonomous Weapons and Human Control: Part I’.

[16] Anthony King, ‘Digital Targeting: Artificial Intelligence, Data, and Military Violence’, Journal of Global Security Studies 9, no. 2 (2024), at: https://doi.org/10.1093/jogss/ogae009; Heiko Borchert, ‘The Very Long Game of Defense AI Adoption: Introduction’, in Heiko Borchert, Torben Schütz and Joseph Verbovszky (eds), The Very Long Game: 25 Case Studies on the Global State of Defense AI (Cham: Springer, 2024), pp. 1–38, at: https://doi.org/10.1007/978-3-031-58649-1; Andreas Graae, Servers Before Tanks? Defence AI in Denmark, DAIO Study 23/18 (Hamburg: Defense AI Observatory, 2023), at: https://defenseai.eu/wp-content/uploads/2023/11/daio_study2318_servers_before_tanks_andreas_graae.pdf.

[17] HW Meerveld, RHA Lindelauf, EO Postma and M Postma, ‘The Irresponsibility of Not Using AI in the Military’, Ethics and Information Technology 25, no. 14 (2023), at: https://link.springer.com/article/10.1007/s10676-023-09683-0; Avi Goldfarb and Jon R Lindsay, ‘Prediction and Judgment: Why Artificial Intelligence Increases the Importance of Humans in War’, International Security 46, no. 3 (2022): 7–50, at: https://doi.org/10.1162/isec_a_00425; Nadibaidze et al., AI in Military Decision Support Systems.

[18] Alex Neads, David J Galbreath and Theo Farrell, From Tools to Teammates: Human-Machine Teaming and the Future of Command and Control in the Australian Army, Australian Army Occasional Paper No. 7 (Australian Army Research Centre, 2021), at: https://researchcentre.army.gov.au/library/occasional-papers/tools-teammates; Neads, Farrell and Galbreath, ‘Evolving Towards Military Innovation’.

[19] Neads, Galbreath and Farrell, From Tools to Teammates.

[20] James Johnson, ‘Automating the OODA Loop in the Age of Intelligent Machines: Reaffirming the Role of Humans in Command-and-Control Decision-Making in the Digital Age’, Defence Studies 23, no. 1 (2023): 43–67, at: https://doi.org/10.1080/14702436.2022.2102486; Goldfarb and Lindsay, ‘Prediction and Judgment’; Scharre, Four Battlegrounds.

[21] Emelia Probasco, Helen Toner, Matthew Burtell and Tim GJ Rudneret, AI for Military Decision-Making: Harnessing the Advantages and Avoiding the Risks (Washington DC: Center for Security and Emerging Technology, 2025), at: https://cset.georgetown.edu/publication/ai-for-military-decision-making.

[22] Goldfarb and Lindsay, ‘Prediction and Judgment’; Johnson, ‘Automating the OODA Loop in the Age of Intelligent Machines’.

[23] Lena Trabucco, ‘AI-Enabled Autonomous Weapons and Human Control: Part II: Human Control and Military Commanders’, International Law Studies 106 (2025): 579–615, at: https://digital-commons.usnwc.edu/ils/vol106/iss1/18.

[24] Mica R Endsley, ‘Supporting Human–AI Teams: Transparency, Explainability, and Situation Awareness’, Computers in Human Behavior 140 (2023), at: https://doi.org/10.1016/j.chb.2022.107574.

[25] Ibid.

[26] Ibid.

[27] Ibid.; Kenneth Payne, I, Warbot: The Dawn of Artificially Intelligent Conflict (London: Hurst Publishers, 2021); Ben Shneiderman, ‘Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy’, International Journal of Human-Computer Interaction 36, no. 6 (2020): 495–504, at: https://doi.org/10.1080/10447318.2020.1741118.

[28] Lena Trabucco and Esben Salling Larsen, Artificial Intelligence in Command and Control (Djøf Publishing and Centre for Military Studies, 2025), at: https://cms.polsci.ku.dk/publikationer/ai-i-kommando--og-kontrolsystemer/Artificial_Intelligence_in_Command_and_Control_download.pdf.

[29] Mary L Cummings, ‘Automation and Accountability in Decision Support System Interface Design’, Journal of Technology Studies 32, no. 1 (2006): 23–31, at: https://doi.org/10.21061/jots.v32i1.a.4.

[30] Endsley, ‘Supporting Human–AI Teams’; National Academies (2021).

[31] Endsley, ‘Supporting Human–AI Teams’.

[32] Raja Parasuraman and Victor Riley, ‘Humans and Automation: Use, Misuse, Disuse, Abuse’, Human Factors 39, no. 2 (1997), at: https://doi.org/10.1518/001872097778543886; Raja Parasuraman and Dietrich H Manzey, ‘Complacency and Bias in Human Use of Automation: An Attentional Integration’, Human Factors 52, no. 3 (2010), at: https://doi.org/10.1177/0018720810376055.

[33] Parasuraman and Riley, ‘Humans and Automation’.

[34] Saleema Amershi et al., ‘Guidelines for Human-AI Interaction’, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (New York: Association for Computing Machinery, 2019), at: https://dl.acm.org/doi/10.1145/3290605.3300233.

[35] Tamsyn Edwards and Paul U Lee, ‘Designing Graceful Degradation into Complex Systems: The Interaction Between Causes of Degradation and the Association with Degradation Prevention and Recovery’, 2018 Aviation Technology, Integration, and Operations Conference (AIAA Aerospace Research Central, 2018), at: https://doi.org/10.2514/6.2018-3510; Michael A Gerber, Ronald Schroeter and Bonnie Ho, ‘A Human Factors Perspective on How to Keep SAE Level 3 Conditional Automated Driving Safe’, Transportation Research Interdisciplinary Perspectives 22 (2023), at: https://doi.org/10.1016/j.trip.2023.100959.

[36] de Sio and van den Hoven, ‘Meaningful Human Control over Autonomous Systems’; Steven Umbrello, ‘Coupling Levels of Abstraction in Understanding Meaningful Human Control of Autonomous Weapons: A Two-Tiered Approach’, Ethics and Information Technology 23 (2021): 455–464, at: https://link.springer.com/article/10.1007/s10676-021-09588-w.

[37] de Sio and van den Hoven, ‘Meaningful Human Control over Autonomous Systems’; Umbrello, ‘Coupling Levels of Abstraction’.

[38] Department of Defense, Autonomy in Weapon Systems, DoD Directive 3000.09. 

[39] Ministry of Defence (MoD), Ambitious, Safe, Responsible: Our Approach to the Delivery of AI-Enabled Capability in Defence (MoD, 2022), at: www.gov.uk/government/publications/ambitious-safe-responsible-our-approach-to-the-delivery-of-ai-enabled-capability-in-defence. 

[40] Australian Army, Robotic and Autonomous Systems Strategy. 

[41] Goldfarb and Lindsay, ‘Prediction and Judgment’; ML Cummings, Artificial Intelligence and the Future of Warfare (Chatham House, 2017), at: www.chathamhouse.org/2017/01/artificial-intelligence-and-future-warfare.

[42] Monica Tatasciore, Luke Strickland and Shayne Loft, ‘Transparency Improves the Accuracy of Automation Use, but Automation Confidence Information Does Not’, Cognitive Research: Principles and Implications 9, no. 67 (2024), at: https://doi.org/10.1186/s41235-024-00599-x; Na Du, Kevin Y Huang and X Jessie Yang, ‘Not All Information Is Equal: Effects of Disclosing Different Types of Likelihood Information on Trust, Compliance and Reliance, and Task Performance in Human-Automation Teaming’, Human Factors 62, no. 6 (2020): 987–1001, at: https://doi.org/10.1177/0018720819862916.

[43] Tatasciore et al., ‘Transparency Improves the Accuracy of Automation Use’.

[44] Ian Henderson and Kate Reece, ‘Proportionality Under International Humanitarian Law: The “Reasonable Military Commander” Standard and Reverberating Effects’, Vanderbilt Journal of Transnational Law 51, no. 3 (2021): 835–855, at: https://scholarship.law.vanderbilt.edu/vjtl/vol51/iss3/12; Emma J Breeze, ‘Duty to Act on Knowledge: Precautions, Intelligence and the Protection of Civilians in Armed Conflict’, Journal of Conflict and Security Law 29, no. 3 (2024): 311–329, at: https://doi.org/10.1093/jcsl/krae015; Trabucco, ‘AI-Enabled Autonomous Weapons and Human Control: Part II’; Renato Wolf, ‘Applying Precautions in Target Verification with AI Decision Support Systems’, Israel Law Review 58, no. 2–3 (2025): 282–305, at: https://doi.org/10.1017/S0021223725100071; Jessica Dorsey, ‘The Erosion of Human(e) Judgement in Targeting? Quantification Logics, AI-Enabled Decision Support Systems and Proportionality Assessments in IHL’, International Review of the Red Cross 107, no. 930 (2025): 1041–1071, at: https://doi.org/10.1017/S1816383125100969.

[45] Department of Defence, Policy Settings.