UK Intelligence Agencies and the Commercial Cloud: What Does It All Mean?

A recent Financial Times report confirmed the procurement of a high-security cloud system by the UK’s intelligence community. The hosting of sensitive intelligence data on non-government systems is a significant pivot away from traditional practice, where the assumption is that in-house capabilities are relied upon to host sensitive data. The immediate concern about data sovereignty is slightly misleading, as it is most likely that the data is stored and processed on UK soil. However, this public-private partnership between the UK government and Amazon does raise questions relating to the UK’s current lack of sovereign cloud capability, incentives for future UK investment in research and development, and existing oversight mechanisms for big tech and cloud technologies.

The Rationale Behind Cloud Technology in a Data-Rich World

The amount of data in the world is doubling every two years and the 21st century has seen an explosion in both openly available data and data that the intelligence services collect and work on. Security agencies must be able to piece these together to deliver on their national security objectives. Intelligence agencies know they will be left behind if they rely on legacy systems to turn data into actionable information, particularly in an environment where adversaries are embracing technology and the analysis of data at scale.

Jeremy Fleming, the director of GCHQ, has stated publicly that ‘China by more or less any measure is doing well in the development of AI capabilities’, while others in the US have gone further to say that China is comfortably winning the ‘AI race’. Innovations in the way that data is held and connected to create a more complete picture of intelligence operations are central to the future of intelligence analysis. The UK must not fall behind in this endeavour.

It is also important to address the assumption that data will automatically be less secure in a cloud environment. In reality, the level of cyber security depends on the risk management controls adopted by a cloud service provider, such as: how network architecture is designed and segmented; the level of investment in innovative security measures; patching regimes; the ability to make ongoing changes to network architecture; training and investment in skilled staff; compliance with regulatory mandates and risk management frameworks; and most importantly, how the data is physically secured. This final point should be emphasised – it is likely in this case that the cloud network will be air-gapped (physically isolated from unsecure networks).

An ‘Off-the-Shelf’ Approach for Security Agencies

RUSI has written extensively about the relevance of AI and machine learning capabilities to achieving security objectives. Integrating AI and machine learning capabilities into the activities of an entity as large as an intelligence agency requires infrastructure that is rapidly scalable, a data centre that can secure and store as much data as it can collect (wherever it collects it from), and computing power that is capable of processing and analysing that data.

Rather than developing that infrastructure in-house, security agencies can choose to rent it ‘off-the-shelf’ from private providers. Although the clientele of a cloud provider is broader than just the intelligence community, by and large intelligence agency requirements are well-served by commercial cloud technology products. From a productivity perspective, it does make far more sense for the intelligence community to use these commercial products to analyse large volumes of data, rather than developing in-house capabilities that may not be fully functional, take far longer to build, require ongoing maintenance, and potentially run well overbudget.

From an analytical perspective, there are clear short-term wins for the intelligence community when using an off-the-shelf capability. Techniques like natural language processing, machine translation and speech recognition can be deployed to reduce processing times, in turn freeing up intelligence analysts’ time to focus on tasks which require more brainpower.

Real-time or near-real-time collaboration across numerous domains and geographical locations can help to create a decision space akin to ‘a single pane of glass’, in the words of one US General. Indeed, we do know that the service being provided by Amazon has interoperability firmly in mind, allowing the intelligence agencies to conduct faster searches on each other’s databases. The positive effects of this on decision-making can be expected to be immediate: a higher-speed, more flexible environment that increases the chances of finding ‘the needle in the haystack’ in a timely fashion. Although the aforementioned techniques can be deployed separately from the cloud, commercial offerings may be lower quality than cloud offerings and the development resource may be prohibitive.

Despite these positives, the use of ‘off-the-shelf’ capability should not be completely at the expense of the UK’s own research and development. One unintended consequence of this partnership could be the slowing down of investment in UK innovation. Looking at EU strategy, there is a strong desire to invest in next-generation tools and infrastructures to store and process its own data. Similarly, vendor diversity matters, and relying on one cloud provider is suboptimal to say the least – though the value for money of contracting multiple providers should also be a consideration.

A Warning for UK Data Sovereignty?

In some quarters, the current deal has been framed as a problem for UK data sovereignty. Some of the arrangements that have been made public, such as the fact that classified UK intelligence data will be stored and protected in the UK and not abroad, and that Amazon will not have any access to information held on the cloud platform, suggest that technically speaking, this risk has been mitigated.

However, there are higher-level questions that do emerge when the UK’s intelligence community gradually becomes reliant on commercial cloud technologies. It is important to recognise that commercial partnerships for the UK agencies are not an overnight development; in particular, the role of US technology companies in UK intelligence infrastructure has been built up over the course of two decades. Thus, the latest Amazon cloud partnership should be seen as the continuation of a long-established trend rather than a pivotal point of departure.

Although it has been reported that the intelligence community attempted to procure this infrastructure from a UK provider, there was no viable UK-based offering. This is clearly a regrettable outcome, but equally a reality check is needed when there are calls for sovereign technology solutions. The funds that would have been required to procure and develop a UK solution would likely have far exceeded the estimated cost of the Amazon contract, and it would have brought a great deal of risk compared to the tried-and-tested services provided by Amazon. Undoubtedly, the UK government would have given these issues a great deal of thought before proceeding with the Amazon deal. More broadly, discussions around sovereign technology are a matter of industrial strategy and lie beyond the remit of the intelligence community.

Future investment in UK innovation means incentivising the private sector to take on more risk with technology investment; introducing incentives for a collaborative model across government, academia and the private sector; and making government the customer of first choice for potential suppliers. There is certainly no quick fix.

What is the Role of Oversight?

The role of oversight mechanisms is usually front and centre of the public debate. But it is important to understand the role that different types of oversight play. In terms of oversight of the intelligence community’s capabilities, mechanisms do exist and should suffice. The cloud services offer an extension of what is currently done in-house but at scale, rather than presenting a whole new set of capabilities. The amount of data that is collected is not necessarily changing, but rather the ability to make sense of that set of data and use it more effectively. The Investigatory Powers Commissioner’s Office has for several years been in place to make judgements on whether the agencies are carrying out their activities in a necessary and proportionate manner. There is no reason for this arrangement to change.

However, there are important oversight-related questions regarding technology companies involved in UK national security. Assessing who is a reliable and trusted partner is a continuous process; even if the company is based in a partner country which is also an intelligence ally, it should still be subject to continuous scrutiny. Much like for the telecommunications sector, there may be existing processes in place for establishing low- and high-risk vendors that are subject to constant re-evaluation.

When national security data is processed by a private company, questions around the performance and auditability of AI systems are understandably emphasised by policymakers. Private companies have shareholders and stakeholders who are answerable to interests that are not necessarily aligned with those of the agencies, and these interests could change over time. Such oversight questions are not new and would have been explored by the Intelligence and Security Committee over the last decade and beyond. However, a quick glance at the Intelligence and Security Committee’s webpage shows that ‘cloud technologies’ have just been listed as the subject of one of its ongoing inquiries. The procurement of cloud technologies by the intelligence community is a live issue.

In the US, Congress is being encouraged to take a more active role in working with industry to create a comprehensive set of rules, guidelines and technology acquisition processes that will make up a healthy data ecosystem. Parliamentary involvement in the UK could help ensure that the cloud services being procured across government meet certain conditions: for example, that interoperability is not impeded by ‘vendor lock’ where capabilities are designed to only work in a single cloud environment. A ‘build once, deploy anywhere’ philosophy should be central to the way that UK government agencies – and not just the intelligence community – are approaching procurement in this space.

Conclusion

As was seen with the reaction to the Palantir/NHS COVID-19 Data Store deal last year, contracts for big technology and data projects are under the microscope. There are undoubtedly higher-level strategic concerns that agreements like the one with Amazon reinforce about the UK’s technology advantage and the ability to bring that to bear when it really matters, as well as the future of in-house innovation in a world where the private sector can offer high-level capabilities.

Yet we must also be clear about the rationale behind the cloud technology revolution and acknowledge the benefits that it will bring not just to the UK, but to adversaries too. Forming effective partnerships with private sector stakeholders will be important to avoid falling behind allies and adversaries, but with that comes a whole set of considerations which both Parliament and the wider public need to engage with in the most informed way possible. For cloud technologies, it seems that a UK parliamentary inquiry intends to do just that.

The views expressed in this Commentary are the authors’, and do not represent those of RUSI or any other institution.

Have an idea for a Commentary you’d like to write for us? Send a short pitch to commentaries@rusi.org and we’ll get back to you if it fits into our research interests. Full guidelines for contributors can be found here.