How AI Is Reshaping Network Requirements in Banking

The Impact of AI on Networking Infrastructure in Finance 

“AI is producing and consuming exponentially larger amounts of data than anything we’ve seen in the past. At the same time, compared with traditional workloads, AI workloads are incredibly fragile,” Gandluru says. “Even the smallest amount of packet loss can cause a training job, one that would take several days to complete, to restart.” 

For organizations with more legacy systems (such as many banks, insurers and payment providers), AI has been introduced alongside existing infrastructure through the cloud. But for various reasons, some companies are starting to repatriate these functions to on-premises data centers, Gandluru says. This is happening without dedicated new teams, and these companies are evolving their network to support traditional applications and AI workloads. 

DISCOVER: How managed cloud and AI services can turn complexity into a business advantage.

Organizations that are more involved in quantitative trading or other highly sophisticated financial services are already operating large-scale AI environments, Gandluru notes. They’re not only trying to fit as much processing power in their data centers as possible to stand out for their customers but also hiring sector-specific AI engineers. 

“There are two main factors that dictate success for these firms: trading strategy and the speed at which they can identify and act on market signals,” he says. 

Understanding Networking Needs Based on AI Workloads 

Gandluru shares four examples of AI workloads that financial organizations may need to account for: 

• Customer-facing applications. An AI mortgage assistant that customers rely on may have peak periods of usage as customers ask questions, compare loan options or upload documents, and these tasks all need to feel instantaneous. “The network needs to deliver consistently low latency while scaling to support a very large number of concurrent AI requests,” Gandluru says. 
• Internal services for employee use. AI features may be woven into nearly every employee workflow, so AI traffic happens throughout work hours. “The challenge is supporting AI as a core resource for the entire workforce to do their jobs better and faster, without impacting other business-critical applications,” he adds. 
• Fraud detection and risk analysis. These workloads are continuous, compute-intensive and highly sensitive, so many financial institutions may choose to run them on-premises for greater control over performance, security, compliance and operating costs. “In these environments, the network's job is to move massive data sets efficiently between storage, graphics processing units and applications without creating bottlenecks,” he says.
• Large-scale AI model training. These workloads typically require the most bandwidth; even small network inefficiencies can slow the entire system. Predictable bandwidth will become increasingly important. “AI performs best when the network delivers consistent throughput, minimal latency and keeps every GPU busy instead of waiting for data. That's what enables financial institutions to move faster, make better decisions and maximize the return on their AI investments,” Gandluru says. 

How to Improve a Networking Strategy for AI 

Customers expect their financial services to be accessible, timely and secure, and organizations are working to meet those demands. At the same time, AI is rapidly evolving: The capabilities that started as ideas are steadily becoming operational realities.

IT leaders in the space cannot wait for the perfect AI strategy, Gandluru says, but they also don’t need to reinvent the wheel. 

“Use validated designs, proven architectures and partners who have already solved many of these challenges. That lets you focus less on integrating infrastructure and more on delivering business outcomes. The organizations that will have the biggest advantage a few years from now won't necessarily be the ones that bought the most GPUs; they'll be the ones that started building the right foundation early,” he says. 

While InfiniBand has been the standard in specialized high-performance computing environments, Gandluru says, Ethernet is becoming the backbone for AI in finance because it can bring scale and accessibility at this stage. 

“It's what most institutions already run, their teams already know it, and the vendor ecosystem behind it is enormous,” he says. “With lossless Ethernet, RoCEv2 (remote direct memory access over Converged Ethernet version 2) and the work coming out of the Ultra Ethernet Consortium, the performance gap is closing fast. That matters because it's what democratizes AI; it lets banks and financial institutions without hyperscaler budgets or specialized InfiniBand teams still build serious AI infrastructure.” 

READ MORE: Find out how your organization can increase the chances of success for AI projects.

Security must be an integral consideration to the network itself, not bolted on as a second thought. Because AI expands the attack surface, identity and access management become even more crucial. 

“Ultimately, the network itself has to get smarter — able to spot issues and adjust on its own — and it needs to be built on open standards so it can keep evolving as AI does, instead of locking institutions into one vendor's roadmap,” he says. 

For years, IT departments have warned about the risks of shadow IT. Similarly, organizations need to be more aware of the risks of shadow AI, especially as companies move from generative AI solutions to agentic AI. 

“Visibility has to come first. Before organizations can govern AI, they have to know where AI is running, what it's connected to, what data it's accessing and what actions it's capable of taking,” Gandluru says. “I think that's going to be one of the defining networking and security challenges of the AI era.”