At Fortune’s recent Brainstorm AI event in San Francisco, an expert roundtable grappled with that question as insiders shared how their companies are approaching security and governance—an issue that is leapfrogging even more practical challenges such as data and compute power. Companies are in an arm’s race to parachute AI agents into their workflows that can tackle tasks autonomously and with little human supervision. But many are facing a fundamental paradox that is slowing adoption to a crawl: Moving fast requires trust, and yet building trust takes a lot of time. 

Dev Rishi, general manager for AI at Rubrik, joined the security company last summer following its acquisition of his deep learning AI startup Predibase. Afterward, he spent the next four months meeting with executives from 180 companies. He used those insights to divide agentic AI adoption into four phases, he told the Brainstorm AI audience. (To level set, agentic adoption refers to businesses implementing AI systems that work autonomously, rather than responding to prompts.) 

According to Rishi’s learnings, the four phases he unearthed include the early experimentation phase where companies are hard at work on prototyping their agents and mapping goals they think could be integrated into their workflows. The second phase, said Rishi, is the trickiest. That’s when companies shift their agents from prototypes and into formal work production. The third phase involves scaling those autonomous agents across the entire company. The fourth and final stage—which no one Rishi spoke with had achieved—is autonomous AI. 

Roughly half of the 180 companies were in the experimentation and prototyping phase, Rishi found, while 25% were hard at work formalizing their prototypes. Another 13% were scaling, and the remaining 12% hadn’t started any AI projects. However, Rishi projects a dramatic change ahead: In the next two years, those in the 50% bucket are anticipating that they will move into phase two, according to their roadmaps. 

“I think we’re going to see a lot of adoption very quickly,” Rishi told the audience. 

However, there’s a major risk holding companies back from going “fast and hard,” when it comes to speeding up the implementation of AI agents in the workforce, he noted. That risk—and the No.1 blocker to broader deployment of agents— is security and governance, he said. And because of that, companies are struggling to shift from agents being used for knowledge retrieval to being action oriented.

“Our focus actually is to accelerate the AI transformation,” said Rishi. “I think the number one risk factor, the number one bottleneck to that, is risk [itself].”

Integrating agents into the workforce

Kathleen Peters, chief innovation office at Experian who leads product strategy, said the slowing is due to not fully understanding the risks when AI agents overstep the guardrails that companies have put into place and the failsafes needed for when that happens.

“If something goes wrong, if there’s a hallucination, if there’s a power outage, what can we fall back to,” she questioned. “It’s one of those things where some executives, depending on the industry, are wanting to understand ‘How do we feel safe?’”

Figuring out that piece will be different for every company and is likely to be particularly thorny for companies in highly regulated industries, she noted. Chandhu Nair, senior vice president in data, AI, and innovation at home improvement retailer Lowe’s, noted that it’s “fairly easy” to build agents, but people don’t understand what they are: Are they a digital employee? Is it a workforce? How will it be incorporated into the organizational fabric? 

“It’s almost like hiring a whole bunch of people without an HR function,” said Nair. “So we have a lot of agents, with no kind of ways to properly map them, and that’s been the focus.”

The company has been working through some of these questions, including who might be responsible if something goes wrong. “It’s hard to trace that back,” said Nair. 

Experian’s Peters predicted that the next few years will see a lot of those very questions hashed out in public even as conversations take place simultaneously behind closed doors in boardrooms and among senior compliance and strategy committees. 

“I actually think something bad is going to happen,” Peters said. “There are going to be breaches. There are going to be agents that go rogue in unexpected ways. And those are going to make for a very interesting headlines in the news.”

Big blowups will generate a lot of attention, Peters continued, and reputational risk will be on the line. That will force the issue of uncomfortable conversations about where liabilities reside regarding software and agents, and it will all likely add up to increased regulation, she said. 

“I think that’s going to be part of our societal overall change management in thinking about these new ways of working,” Peters said.

Still, there are concrete examples as to how AI can benefit companies when it is implemented in ways that resonate with employees and customers. 

Nair said Lowe’s has seen strong adoption and “tangible” return on investment from the AI it has embedded into the company’s operations thus far. For instance, among its 250,000 store associates, each has an agent companion with extensive product knowledge across its 100,000 square foot stores that sell anything from electrical equipment, to paints, to plumbing supplies. A lot of the newer entrants to the Lowe’s workforce aren’t tradespeople, said Nair, and the agent companions have become the “fastest-adopted technology” so far.

“It was important to get the use cases right that really resonate back with the customer,” he said. In terms of driving change management in stores, “if the product is good and can add value, the adoption just goes through the roof.”

Who’s watching the agent?

But for those who work at headquarters, the change management techniques have to be different, he added, which piles on the complexity. 

And many enterprises are stuck at another early-stage question, which is whether they should build their own agents or rely on the AI capabilities developed by major software vendors. 

Rakesh Jain, executive director for cloud and AI engineering at healthcare system Mass General Brigham, said his organization is taking a wait-and-see approach. With major platforms like Salesforce, Workday, and ServiceNow building their own agents, it could create redundancies if his organization builds its own agents at the same time. 

“If there are gaps, then we want to build our own agents,” said Jain. “Otherwise, we would rely on buying the agents that the product vendors are building.”

In healthcare, Jain said there’s a critical need for human oversight given the high stakes. 

“The patient complexity cannot be determined through algorithms,” he said. “There has to be a human involved in it.” In his experience, agents can accelerate decision making, but humans have to make the final judgment, with doctors validating everything before any action is taken. 

Still, Jain also sees enormous potential upside as the technology matures. In radiology, for example, an agent trained on the expertise of multiple doctors could catch tumors in dense tissue that a single radiologist might miss. But even with agents trained on multiple doctors, “you still have to have a human judgment in there,” said Jain. 

And the threat of overreach by an agent that is supposed to be a trusted entity is ever present. He compared a rogue agent to an autoimmune disease, which is one of the most difficult conditions for doctors to diagnose and treat because the threat is internal. If an agent inside a system “becomes corrupt,” he said, “it’s going to cause massive damages which people have not been able to really quantify.”

Despite the open questions and looming challenges, Rishi said there’s a path forward. He identified two requirements for building trust in agents. First, companies need systems that provide confidence that agents are operating within policy guardrails. Second, they need clear policies and procedures for when things will inevitably go wrong—a policy with teeth. Nair, additionally, added three factors for building trust and moving forward smartly: identity and accountability and knowing who the agent is; evaluating how consistent the quality of each agent’s output is; and, reviewing the post-mortem trail that can explain why and when mistakes have occurred. 

“Systems can make mistakes, just like humans can as well,” said Nair. “ But to be able to explain and recover is equally important.”

Hello and welcome to Eye on AI. In this edition….Insights from Fortune Brainstorm AI San Francisco…Disney invests $1 billion in OpenAI and licenses its IP to the company…OpenAI debuts GPT-5.2 in effort to silence concerns it’s trailing rivals…Oracle stock takes a tumble.

Hi, it’s Jeremy here. I’m still buzzing from Fortune Brainstorm AI San Francisco, which took place earlier this week. We had a fabulous lineup including Brad Lightcap, OpenAI’s chief operating officer, Google Cloud CEO Thomas Kurian, Intuit CEO Sasan Goodarzi, Exelon CEO Calvin Butler, Databricks CEO Ali Ghodsi, Rivian CEO RJ Scaringe, Insitro CEO Daphne Koller, and many more. We also had a thoughtful conversation on AI’s impacts with actor, director, and increasingly AI thought leader Joseph Gordon Levitt, as well as a scream of a session with actor, comedian and AI CEO Natasha Lyonne. Today, Sharon Goldman, Bea Nolan, and I are going to share a few highlights and personal impressions.

For me, there was a notable vibe this year that a lot of companies are substantially further along in implementing AI across their organizations, including using AI agents in some limited, but important, capacities. Many audience questions, especially in some of the breakout sessions, were around governance and orchestration methods for an increasingly hybrid workforce where AI agents will be completing tasks alongside employees.

Still, it was striking to hear Butler, the Exelon CEO, say that his company is moving cautiously. When the consequence of getting something wrong is literally lights out, security and reliability have to take precedence over everything else. And so Butler said he was happy not to be a “first mover” but instead a “fast follower” when it came to AI implementations. Let other people take the hit and learn from their mistakes, seems to be his view.

And this wasn’t the only place where speakers were seeking to tamp down hype. It was refreshing to hear Michael Truell, the cofounder and CEO of hit coding assistant Cursor tell me that he didn’t think software engineering would ever be fully automated in the way that OpenAI CEO Sam Altman sometimes talks about. Instead Truell said that while the amount of time that coders spent on “compilation” of code would continue to shrink, he saw a continued need for humans to make design decisions around “how should the software work.”

Similarly, Vidya Peters, from DataSnipper, said she thought there would still be a role for qualified accountants within finance organizations, even if they were increasingly being assisted with AI tools such as the one her company makes. She also said she thought that applications geared specifically for a particular industry or job—especially in regulated industries—would continue to win out over more general purpose AI models, even as the big AI companies are increasingly targeting specific professional use cases for their general purpose models.

A panel that Sharon moderated on the “new geography of data centers” was fascinating. The message was that right now, data centers are going where the power is. But increasingly data centers are going to be looking to build their own power on site and possibly even become net contributors to the grid. And Jason Eichenholz, the CEO of Relativity Networks, said that as AI inference workloads come to eclipse AI training workloads, there will be an increasing need to bring data centers close to major population centers, but that most cities in the U.S. are power constrained. How are we going to get these urban centers the tokens they need at the speed at which they need them? That’s anyone’s guess right now, Eichenholz says—although his company builds the fast fiber that will carry those tokens from the data centers to end users.