CJ Moses, the Chief Information Security Officer and VP of Security Engineering at Amazon , has been at the company for 17 years but rarely speaks publicly about the company’s security.

So, when he does, people listen.

In an interview with ITWire at AWS’ re:Invent 2024 event in Las Vegas, Moses acknowledged that  “we didn’t think there was the need to talk publicly about what we were doing. But in the past six or eight months our customers have been asking because others have been talking and chest pumping and our customers are saying ‘why aren’t you talking about this. I am sure you are doing something’. The reality is that, firstly, our company culture is a bit more humble, secondly, the less out there about how we do things and how we defend ourselves the less the adversaries know.”

Moses said that behind the scenes he is regularly in contact with customers about what the company is doing to keep their data safe. “Customers increasingly ask us where our threat intelligence comes from, what types of threats we see, how we act on what we observe, and what they need to do to protect themselves.”

AWS operates one of the most trusted cloud infrastructure on the planet. Organisations around the world place a lot of trust in AWS to look after their most sensitive data. And it is no wonder.

Moses said the number of threats the company detected through its honeypot system, MadPot, was around 100 million a day eight months ago. After making further enhancements to MadPot, that number is now around 750 million a day and continually on the rise. The massive increase is not only a reflection of increased hacker activity, but AWS is also detecting more threats using the smarts of Gen AI.

“Every day across AWS infrastructure, we detect and thwart cyberattacks. With the largest public network footprint of any cloud provider, AWS has unparalleled insight into certain activities on the internet,” he says.

“Not only is our threat intelligence used to enrich security services that AWS and our customers rely on, we also proactively reach out to share critical information with customers that we believe may be targeted or potentially compromised by malicious actors. Sharing our threat intelligence enables recipients to assess information we provide, take steps to reduce their risk, and help prevent disruptions to their business.”

Moses said AWS is constantly looking to improve its ability to see and react to actors’ evolving tactics, techniques, and procedures. To this end they created MadPot, a threat intelligence tool used to protect customers from cybercrime.

“We have created large-scale systems so we can see different vulnerabilities. We need to see what is getting through and what isn’t,” Moses admits.

“That threat intelligence informs other systems within Amazon to be able to protect our customers. On average we are seeing 750 million hits a day via MadPot. We get that information in as close to real time as we can and putting it into other systems that can do something about the threats that we see. We have a system, Sonaris, which allows us to block and tackle a lot of the things we see out of MadPot and other systems.”

Moses also spoke about why AWS had decided to create its own security system rather than rely on third parties.

“There is a lot of third-party threat intelligence and we often get asked why doesn’t all of that work for us and why do we have our own?

“The answer is pretty simple. Over a three-minute period in the AWS environment we will have a 23% change in IP addresses. We are that dynamic that if we get IP address-based threat intel that is an hour old the likelihood is that that threat intelligence is not very useful to us. The days of tracking metrics to mean time to detection are inadequate as well because if we put a new MadPot sensor out there with a new vulnerability within 90 seconds it will have been scanned and within three minutes attacked.

“If you do the math - remembering the 750,000 million hits a day – that dynamic nature of the threat means that we have to be responsive so much more quickly. We don’t track mean time to detection. The meaningful metric for us is mean time to defence. That is the metric we care about. If you want to focus on something you track a metric to it not to part of it.”

Moses said MadPot identifies the intel, Sonaris and other systems (such as Guard Duty) will get that data and the meantime defence metric is the one it is looking for because “we need to be shorter than the attack metric. It has to be within minutes, seconds in some cases to be able to be effective.

“The reliance on third party threat intel for that type of model doesn’t work for us.  That doesn’t mean we don’t have good partnerships where identification of the threat and some of the other indicators they provide we aggregate and work really closely with our community on that. But on a day-to-day basis we have to automate and that has to be faster than humans and be responsive to the needs of our customers.”

Moses said AWS received and analysed thousands of different kinds of event signals in real time. “For example, MadPot observes more than 100 million potential threats every day around the world, with approximately 500,000 of those observed activities classified as malicious. This means high-fidelity findings (pieces of relevant information) produce valuable threat intelligence that can be acted on quickly to protect customers from harmful and malicious online activities. Our high-fidelity intelligence also generates real-time findings that are ingested into our intelligent threat detection security service GuardDuty, which automatically detects threats for millions of AWS accounts.”

All these capabilities are also a benefit to AWS and Amazon. “Internally we have 1.7 million employees that we need to be able to protect and that same system is protecting all of the Amazon infrastructure because it is mostly customer associated,” Moses says.

“We are able to identify hundreds of thousands of domain derogatory or nefarious domains on a daily basis to be able to block and tackle as appropriate.

“We are about stopping bad things happening to good people. But for nearly 17 years we didn’t say boo about threat intel.”

Moses admits that opening up more about what happens behind the scenes at AWS is because “customers want to know, at least, generically that we are looking out for them and what we are doing and that is what I think more broadly why we are talking more about this.”

Moses says that “back in the old days it was all about Malware. But now it is about identity. Identity is the attack vector of this century or at least this decade.”

“AWS is the most secure cloud provider in the world. I know because I help build it with a great team.”

Moses is an interesting character himself. Away from work he is an active SRO GT America GT2 race car driver. He is a former FBI agent and served as a Special Agent with the Air Force Office of Special Investigations. He was working for the FBI - where he led technical analysis of computer and network intrusion efforts - when the FBI approached AWS about becoming a customer.

“We had a mission of taking every piece of the US government data about anti-terrorism efforts and cross correlating against everything we know. AWS had recently launched its EC2 virtual server and we wanted to use its processing power.”

Moses, along with a handful of others, moved to AWS.

In his current role Moses leads security engineering and operations across Amazon. His mission is to enable Amazon businesses by making the benefits of security the path of least resistance. He joined Amazon in December 2007, holding various roles including Consumer CISO, and most recently AWS CISO, before becoming CISO of Amazon in September of 2023.

Moses spoke at some length about threat hunters – the IT professionals searching for threats before they can be detected by automated systems or traditional security measures.

“We have this broad scale system that is doing all this stuff and we have a lot of Gen AI but the reality is we are never going to replace smart humans. The smart humans need to look for the things that Gen AI may call out - that something it has seen is out of the normal, an apparition. They need to be asking: “I don’t know what it is, but this is not normal.” There is also the aggregation of a bunch of information that doesn’t make sense or are patterns – that is where our threat hunters come in, although we don’t call them threat hunters as they are engineers so we call them threat engineers. They get down into the ones and zeros on this stuff. They will do threat hunting using alerts and those indicators based on the large-scale systems.

“We can never have enough threat engineers looking at all the things we are doing.”

The other side also occurs. “We get constant threat intel that comes from our partners, friends and the community. They will come to us and say ‘I saw this and they will give us an IP address’. I can’t block anything as it is likely that the IP address has changed but what I can do is use our intel to go back and figure out what entity had ownership access to this, what did they do.”

Moses, as you could expect, says he believes his team is the best in the world.

“My focus is on hiring higher level threat engineers to build them into being the experts that they are.  Above that we hire from the intelligence community in places where they have great expertise to be able to bring that forward.

“They will tack 300 to 400 threat actors on a regular basis. We are not tracking a lot in great detail. We need to prioritise them.

“Our threat engineers are getting better, we are getting better. We have also enhanced our algorithms so we find more.”

Moses says that with MadPot and Sonaris they had blocked 2.7 trillion attempts in the past 12 months just in the EC2 environment.