Hybrid Cloud Strategy for AI: Why Workload Placement Determines AI ROI

What Is a Hybrid Cloud Strategy in the Context of AI?

Dave McCarthy, vice president of cloud and edge infrastructure services at IDC, says in enterprise AI, a hybrid cloud strategy is far more than a tactical combination of on-premises hardware and public cloud instances.

“It serves as a deliberate, integrated architectural framework designed to harmonize data flows, model training, fine-tuning and inference across private facilities, public clouds and edge endpoints,” he explains.

Rather than treating these environments as isolated silos, a true hybrid strategy establishes common data planes, unified security protocols and containerized workload mobility.

This allows an organization to dynamically match each phase of the AI lifecycle to the infrastructure environment that best optimizes performance, security, compliance, latency and cost.

Why Many Organizations Run AI in Hybrid Environments Without a Strategy

Robert Daigle, director of global AI business at Lenovo, says over the past two years, organizations have prioritized pilots and proof-of-concepts, often deploying AI wherever infrastructure and data were readily available rather than following a cohesive, long-term plan.

This “build fast” approach has helped validate use cases, but it has also introduced unintended consequences ranging from fragmented architectures to inconsistent governance.

“Without a clear strategy, these environments quickly accumulate technical debt, leading to suboptimal designs, duplicated infrastructure and escalating costs,” Daigle says.

In effect, many organizations have arrived at hybrid AI by necessity, not design.

The next phase of AI maturity will require a shift from experimentation to intentional architecture, where strategy, governance and workload placement are aligned from the outset to avoid cost overruns, reduce risk and enable scalable value creation.

DIVE DEEPER: Learn what you’ll need to build a foundation for scalable AI.

Workload Placement in the AI Era: On-Premises vs. Cloud vs. Edge Decision Framework

Murali Gandluru, vice president of product management, data center networking at Cisco, says workload placement decisions must be driven by workload intent and its specific requirements.

“The edge is essential for real-time inference where latency is a dealbreaker, such as autonomous systems on a factory floor,” he explains.

On-premises environments are often the best choice for training models on massive, proprietary datasets where data gravity makes moving information to the cloud prohibitively expensive or risky.

He adds that the public cloud remains the ideal venue for burst capacity, rapid prototyping and accessing specialized AI services that are not feasible to maintain in-house.

“The leadership challenge is ensuring these choices don’t lead to ‘forklift upgrades’ every time a workload needs to scale,” Gandluru says.

Building a Flexible Governance Framework for Distributed AI Pipelines

Pravjit Tiwana, senior vice president and general manager of cloud storage and services at NetApp, says the most successful organizations separate governance from infrastructure.

“In the past, governance was often tied to a specific environment,” he notes. “In AI, that approach breaks down because workloads, models and data are inherently distributed.”

From his perspective, governance needs to become policy-driven, data-centric and portable. Organizations need consistent controls for lineage, access, retention, sovereignty and model usage regardless of whether a workload runs on-premises, in a public cloud or at the edge.

McCarthy says that to build a flexible framework, organizations must transition from rigid centralized gatekeeping to automated, policy-as-code guardrails embedded directly into CI/CD pipelines.

“This approach balances innovation and control by establishing global compliance baselines such as data lineage and bias checks while granting localized teams the autonomy to execute within those boundaries,” he explains.

DISCOVER: How to optimize your organization’s infrastructure for AI.

Managing Cost Visibility and Resource Allocation Across Multi-Cloud AI Environments

Tiwana says the organizations that win won’t necessarily be the ones that spend the most, but rather the ones that can move workloads to the most efficient environment at any point in time.

“That requires visibility not only into model costs but also into storage, networking, data movement and infrastructure utilization,” he says.

He points out that customers are asking for the ability to run AI workloads wherever economics, performance and governance are most favorable, without having to rebuild their pipelines.

“That’s one reason we believe portability across AI infrastructure will become increasingly important,” Tiwana says.

Gandluru says organizations must move beyond basic billing dashboards to full-stack observability, where IT leaders can correlate GPU utilization, NIC metrics, optics health and job-level telemetry — monitoring everything from GPU usage to data egress fees in one place.

“That correlation lets you make data-driven decisions about where workloads should run and catch performance bottlenecks before they hit application throughput,” he says.

WATCH: Find out how Cisco’s latest offerings are improving observability in the agentic era.

Security and Compliance in Hybrid AI

Daigle says AI is forcing a fundamental shift away from perimeter-based security. In a world where data, models and AI agents operate across edge, data center and cloud, the traditional notion of a fixed boundary no longer applies.

“Security and compliance must now be policy-driven, data-centric and continuously enforced across a highly distributed environment,” he explains.

Equally important is security at the model and training level. Organizations must ensure that training data is governed, models are validated and outputs are aligned with enterprise policies.

This includes rigorous review processes, compliance checks and centralized oversight frameworks to manage risk across the AI lifecycle.

From Lift-and-Shift to Intentional Architecture: A Roadmap for IT Leaders

McCarthy says IT leaders can move past basic lift-and-shift tactics by designing architectures around specific workload intents, mapping data dependencies, latency needs and compliance limits before placing compute.

“This transition is enabled by standardizing on cloud-native containerized platforms and a unified data fabric to ensure seamless, cost-effective portability across environments,” he says.

Daigle advises organizations to focus on placing workloads where they create the most value. This requires leaders to evaluate workloads based on performance, compliance and cost requirements rather than defaulting to a single environment.

“The most successful organizations are taking a workload-first approach by building flexible hybrid architectures that allow them to run the right workload at the right time,” he says.