Q&A: How Capital One Uses Serverless to Move Faster With Less Ops Overhead

BIZTECH: How has that affected the way your team develops?

McNAMARA:  In general, our teams see a reduction in what we term run-the-engine work — things like server patching. One of the things that feature teams would traditionally be responsible for would be infrastructure management. Are we running the latest versions of operating systems? Are we patching everything? Well, that's a whole class of work that simply doesn’t have to happen.

So, instead, you can really focus on driving customer value. When you pull that infrastructure management out altogether, you find that you can really get more innovative with the work that you do.

I work with a lot of teams that have seen the ability for much faster iteration of features that are on their backlogs or are able to do more with less, where much smaller teams can in fact deliver the same value. Some teams have seen up to a 30% drop in their run-the-engine tasks. So, then they’re able to deliver that much more with the same head count or potentially do more with less.

UP NEXT: How banks are using AI for help with regulatory compliance.

BIZTECH: What are the biggest points of friction in your experience for developers?

McNAMARA: Historically, teams struggled with replicating a cloud environment locally or knowing what tools were available and how they ought to use them. Because we are operating in a regulated industry, obviously security compliance and governance are really important. So, when it comes time to deploy, as you might imagine, our deployment pipelines have pretty rigorous security and compliance checks.

One thing we have done is shift as much of the security and compliance checks to the left. So, developers don’t have to wait a long time to find out, “Oh, geez, I tried to deploy a thing and the thing that I tried to deploy didn’t conform to Capital One’s security or compliance standards.”

Over the past few years, we’ve worked really hard to ensure that we’ve shifted that left. So, developers know early on, “Hey, if I try to push my serverless function or serverless application into production, how confident am I that it’s actually going to go through the release process, so those minutes won’t be wasted because of a security or compliance miss?”

BIZTECH: At Capital One, you’re said to be “developing for comprehensive observability from the outside.” What do you mean by that?

McNAMARA:  One thing I hear from time to time is that serverless is “no ops.” And that makes me cringe a little bit because observability is a necessary part of operations. Like, you shift what you ask your provider to do, but ultimately, you’re on the hook. As a developer on these platforms, it’s up to you to make sure that you understand what your application is doing. With serverless, and AWS Lambda in particular, you have to be way more disciplined than you might ordinarily be in a containerized environment or in, say, an instance-based environment. That’s because you can’t cheat.

You know, you’re dealing with ephemeral environments that don’t provide the ability to log in, right? So, you can’t log in to your Lambda function and see what's happening. You can’t run typical shell commands that you normally might. How many processes are running? What's my CPU doing? How much memory is available? Disk space? And on and on. You lose that.

What that means is that as a developer, you have to bake a lot of things in up front because you can’t cheat. It’s crucial to make sure that you’re logging the things you ought to. Conversely, you don’t want to put noise in your logs either. You want to make sure that your logs are structured. So, if you are using other tools like, say, AWS CloudWatch Logs Insights or Splunk or other services like that, your querying becomes a lot easier.

READ MORE: How data is being used to accelerate financial innovation.

BIZTECH: What other tools are making life easier for developers?

McNAMARA:  Metrics are another one. With any AWS service, you get a number of metrics out of the box. In the case of Lambda, you get things like: How many times am I invoking my Lambda function? How long is it taking? Things like that. And as a developer, you also have the ability to create your own custom metrics. So, if you want to capture business-level metrics, user signup, you can easily do that. AWS provides a lot of functionality in some of their modules to make this trivial. We’re big believers in a series of libraries that AWS provides called AWS Powertools. Essentially, it’s a series of modules that makes it easy to do a lot of these observability things, including things like custom metrics, structured logging and tracing.

If you're looking at observing distributed applications, tracing becomes really important. So, metrics and logs are really good for localized understanding, what's happening in my one unit.

Traces let you span multiple things. So, when it comes to tracing, you have the ability to enrich those traces as well. You can get a lot out of the box, but you can also add business-level metadata to these traces. We think that, especially when you're dealing with a large number of microservices, having that insight becomes really important when you need to not only understand what’s happening in your day-to-day, but when things go sideways.

BIZTECH: How are the tools you’re using at Capital One helping you break down silos in the development process?

McNAMARA: The DevOps movement is really about harmonizing developer and operations practices. To me, it’s not really oriented around tools. But having said that, you clearly need tools when you're working with technologies. For me, and I think for Capital One in general, we try to create rails for teams. Regardless of your experience with different technologies, we try to make it easy to deploy units of things, whether it’s AWS Lambda functions, Fargate tasks or DynamoDB tables, we try to set up rails so that teams can operate safely within those rails.

We also make sure that we have those observability tools so that when things are going well and you just want to see how your application is performing over time, great. When they’re not going well, you have the tooling needed to troubleshoot what’s happening. All of which is to say, with serverless technologies in particular, I think you see your cloud provider handling what would have been traditional ops.

DISCOVER: Turn data into insights and accelerate artificial intelligence initiatives.

BIZTECH: How has artificial intelligence affected the developers you're working with?

McNAMARA: At Capital One, we view data as our lifeblood. It really is so critical to what we do and the value that we deliver to our customers. November 2022 was a landmark, Apollo moon-landing moment with the release of ChatGPT. But really, we’ve been using AI and machine learning throughout. When it comes to DevOps, we do use AI to optimize our own cloud operations, and when it comes to things like incident management or incident detection, AI helps us get to resolution faster.

And really, the other key area that we’re using AI in is threat modeling. Working in a regulated industry, we value the ability to detect threats and remediate threats as quickly as possible.

BIZTECH: How is AI helping you do that?

McNAMARA: If we look at the amount of data that we capture, obviously it’s not just localized to our customer interactions. We capture a tremendous amount of telemetry data for what's happening within our systems themselves. We’ve put systems in place that can help us capture log data, metric data and trace data — that’s all being aggregated. So, when bad things happen, from an operational standpoint, our engineers have the ability to use that information to quickly diagnose issues and move to resolution.