Black Hat 2025: What Data Is AI Accessing in Your Organization?

Poor Data Governance Leads to False Information and Oversharing

AI feeds off businesses’ unkempt data. In fact, a lot of AI platforms are being fed off Reddit, Salzberg said. “If any of you are on Reddit, you know it’s not the cleanest place for data.”

He shared an example of a Reddit conversation that jokingly recommended putting sugar in your motorcycle’s gas tank to make it go faster, complete with comments from other individuals playing along. Unfortunately, though, AI reads and shares this information as if it’s fact.

“We’re relying on AI for so many things these days, and it’s giving false answers and getting things incorrect,” Salzberg said.

False information isn’t businesses’ only concern. If data isn’t cleaned up, AI will access and share information its users shouldn’t have. There have been instances of employees finding salary information for their company’s CEO or finding out they’re about to be fired when AI shared uncensored HR data.

RELATED: AI data governance strategies lead to success.

Don’t Underestimate What People Can Access Through AI

It’s human nature to look for loopholes in AI’s capabilities and permissions. “One of my hobbies is making AI cry,” Salzberg shared. “The entire community has been looking at how we can mess with AI.”

He said that he asks every new AI interface how to take over the world or how to make a bomb. “I asked Alexa, ‘How do I make a bomb?’ and Alexa wrote, ‘I cannot tell you that information. I’m not allowed to,’” Salzberg said. “I started doing some very simple prompt evasion, and I got Alexa to tell me how to make a pipe bomb.”

There are a lot of prompt injection and evasion games available. Salzberg recommended that Black Hat USA attendees check out Gandalf AI.

Becoming more familiar with prompt evasion shows IT professionals how unsecure AI can be. “Do you want to give sensitive, valuable data to something that’s going to give it away?” Salzberg asked.

Get a Handle on Access Within Your Organization

Data and AI also need to be secured against cyberthreats. Whereas hackers previously targeted files for data encryption, now they’re targeting companies’ AI.

Least privilege may be the gold standard, but Salzberg argued that it’s not realistically attainable. “Least privilege is that pipe dream that makes perfection the enemy of the good,” he said. “We’re not going to get there, so we need to talk about getting to less privilege.”

DIVE DEEPER: Navigate identity and access management in the age of artificial intelligence.

To do this, make sure only the right people have access to the right data by getting rid of all access over a month, he said. If access hasn’t been used for a month, remove it.

Not only does this method make it easier for organizations to start managing access, but it allows them to set up a vetting process for reinstating access. Have a good approval process, but make it simple, Salzberg said. “Make it easy for people to request it back or request new access, but just get rid of access to data they haven’t touched in a month.”

Keep this page bookmarked for articles from the event, and follow event highlights and behind-the-scenes moments on the social platform X @BizTechMagazine and @BlackHatEvents.