What It Takes To Run AI in Production

BIZTECH: What separates organizations that successfully run AI in production from those that get stuck in the pilot phase?

WEISS: There are a few table stakes. You need governance. You need observability. You need cost controls and the ability to manage the environment as it grows. None of that is particularly glamorous, but it’s essential for running production AI.

The other area that doesn’t receive enough attention is evaluation. An AI system has to be continuously tested to ensure it’s doing what it’s supposed to do. That’s not something you do just once before deployment. It has to happen throughout development and after the system is in production.

Traditional software is relatively deterministic. If you write a unit test, you generally expect the same result every time. AI doesn’t work that way. The same prompt can produce slightly different outputs, so evaluating quality becomes more challenging — and much more important.

The organizations doing this well also invest just as heavily in change management as they do in technology. Too many companies deploy AI tools without thinking about how employees’ day-to-day work will change or how they’ll build confidence in using them. If people don’t embrace AI as part of the organization’s DNA, you’ll end up with underutilized tools. At that point, it doesn’t matter how good the technology is.

The best AI systems don’t replace people. They augment people’s capabilities and increase the speed and quality of their work.

BIZTECH: What capabilities should organizations have in place before they try to scale AI across the business?

WEISS: One of the biggest decisions is choosing a platform. Some organizations want to support every possible AI ecosystem, and that can work. But it also creates fragmentation. Employees don’t know where to go, different teams build in different environments and eventually someone has to clean up the complexity.

Every organization should make decisions based on the platforms they already use. If you’re primarily a Google Workspace organization, Gemini may be the natural choice. If your workflows align better with Anthropic or another ecosystem, build there. The important thing is to commit to a platform instead of scattering AI across dozens of disconnected environments.

Organizations should also take advantage of the capabilities that already exist within those ecosystems. Many companies immediately start building highly customized agents when off-the-shelf tools can already accomplish much of what they need. Instead of recreating general-purpose capabilities, they should extend the tools that already exist with custom skills that solve the organization’s unique problems.

BIZTECH: How important is AI literacy? Today’s tools are becoming easier to use, so do organizations still need formal training?

WEISS: The technology is definitely becoming easier to use, but AI literacy isn’t really about learning complicated software.

It’s about creating curiosity.

You want people to develop the mindset of asking, “I wonder if AI could do this?” Once employees begin experimenting, they often discover entirely new ways of working on their own. That’s one of the most exciting things about AI today. You may be the first person who’s ever tried a particular workflow or solved a specific problem that way.

Some people embrace that sense of discovery immediately. Others are uncomfortable because they want validation before trying something new. That’s where organizations still need change management. When someone opens a conversational AI interface for the first time, they’re often staring at an empty prompt box wondering, “What am I supposed to do with this?” You have to help people get over that hurdle.

Traditional training still matters. Workshops, demonstrations, videos and peer coaching all play an important role. Organizations shouldn’t assume employees will automatically understand what a custom AI agent is designed to do or how it fits into their daily work. Technology alone doesn’t create adoption.

GET THE DETAILS: Build a data infrastructure that supports AI initiatives.

BIZTECH: Once an AI application reaches production, is it ever really finished?

WEISS: Absolutely not.

I have a saying: Great software is never done, but bad software usually is. Whenever someone tells me a software project is finished, I get nervous because that’s often when things start going in the wrong direction. AI is software, and software requires ongoing maintenance. That includes security updates, model upgrades, performance optimization and continuous testing.

Models are evolving every few months. If you declare an AI project finished, there’s a good chance it will stop working properly as underlying models are updated or deprecated. Even when newer models perform better overall, they don’t behave identically. Each model has its own personality and follows instructions a little differently.

Run the same prompt through OpenAI and Anthropic models and you’ll often receive two different — but equally valid — responses. If you’re operating a production AI system, you need to evaluate how those changes affect your application before switching models.

Maintaining production AI isn’t just about keeping systems online. It’s about continuously balancing cost, latency and performance while taking advantage of improvements across the AI ecosystem. If you don’t, you’ll quickly fall behind.

Click the banner below to learn how AI is changing the cybersecurity landscape.

BIZTECH: What’s the right balance between strategic AI initiatives and grassroots experimentation across the organization?

WEISS: I think organizations need both a top-down strategy and a bottom-up strategy. The top-down approach focuses on the biggest business opportunities. Leadership identifies a handful of strategic initiatives where AI can make a meaningful impact and invests accordingly.

Every organization also has thousands of smaller workflows that only the people doing the work truly understand. That’s where the bottom-up strategy comes in.

We want employees to learn how to build agents themselves. We want them to take the processes they know better than anyone else and ask how AI could improve them. I want building an AI agent to become as natural as creating a PowerPoint presentation.

When employees have that capability, they stop waiting for IT to solve every problem. They begin creating solutions for the small, repetitive tasks that consume their own time every day. Those individual improvements may seem minor, but across a large organization they can add up to thousands of specialized agents solving thousands of microproblems.

That’s how AI becomes embedded throughout the business rather than existing as a handful of isolated enterprise projects.

UP NEXT: Learn why it’s so difficult to measure the return on your artificial intelligence investments.

BIZTECH: What’s your advice for organizations planning major AI investments over the next couple of years?

WEISS: Don’t think about AI as a single project. Think about building an organizational capability.

Choose a platform your employees already use. Invest in governance and evaluation from the beginning. Help people become comfortable experimenting with AI instead of expecting every innovation to come from a centralized team. At the same time, identify a few strategic initiatives where AI can create meaningful business value and invest heavily there.

The organizations that succeed will combine those two approaches.

Leadership will establish the vision and provide the right tools, while employees throughout the organization will continuously discover new ways to apply AI in their own work.

That’s ultimately how AI reaches every corner of the enterprise. It’s not through one transformational application, but thousands of smaller improvements that, together, fundamentally change how work gets done.

DISCOVER: Turn data into insights and accelerate artificial intelligence initiatives.