Falling behind: Big data and the analysis race

'Big data' is not a new concept. In fact, it is the processing of big data that determines your suggested books on Amazon, or the advertisement that plays before your YouTube video. Digital corporations such as Google and IBM have been harnessing big data for much longer than mainstream academia has been discussing it, the latter for at least two decades. Big data is, in its simplest form, the analysis of extremely large data sets to reveal patterns, behaviour and trends, especially in a human behavioural sense. So why has it only been noticed by government institutions and think-tanks over the last two years?

In 2013, DARPA Director Regina Dugan told US Congress that 'detecting dismounted fighters in Afghanistan required collecting and processing about 100,000 times more data than it takes to detect a strategic bomber'. By international standards, the population of Afghanistan has an internet penetration rate of 5.5 per cent, placing it in the bottom 10 per cent of connected countries in the world. This figure reflects urbanised areas of the country, not that part where the majority of fighting occurred in the ten-year conflict. There is little disagreement that conflict in Afghanistan was largely rural and digitally disconnected, but is unlikely, given current connectivity trends, that Australia will fight in such a disconnected space again.

What about the future that is described in the Future Land Warfare Report – the urban littoral operating environment? Does Army have the processing power required to store and, perhaps more importantly, analyse digital information in the most densely connected part of the globe?

David Kilcullen recently noted that connectivity has grown so quickly over the last decade, that the Asia–Pacific region now accounts for more than half of the mobile-cellular subscriptions across the globe (3.5 billion out of 6.8 billion). The worldwide connectivity boom has brought with it a number of significant challenges for both government and law-enforcement agencies across the globe, particularly in the tracking, collecting and storing of data. Realisation of the potential of big data analytics is only beginning, particularly as more interest and subsequent investment is stored in practical systems like IBM's Watson or i2. For Army, this means that the effective analysis of open-source information may identify patterns of behaviour and therefore support ground troops, enhance planning and facilitate the propagation of lessons in real-time.

The presence of digital mechanisms within the battlespace for both combatants and civilians will increase in time, but the intelligence edge on the battlefield will only be gained with dynamic storage and effective analysis of information. If the enemy can deliver orders and expedite movement over mobile phones faster than you can analyse and interpret that information, what then is the point of analysing that information in the first place? And that was only one phone call. Development and integration of automated data analysis systems needs to occur before it's too late, particularly as the proliferation and subsequent use of commercial technology outpaces that of government agencies.