AI Risks Are Expected To Hit the Financial Sector Hard

And while the participating experts had a grim outlook that predicted a majority of the risks had some likelihood to lead to “catastrophic outcomes,” they also outlined action within the report that could reduce or eliminate these risks: “to shore up defenses in critical sectors, to ensure actors are held responsible for the risks they own and to address structural dynamics that persist, even assuming pragmatic mitigations are in place.”

“We’re not saying these things are definitely going to happen. We’re saying these are the things that experts think are worth paying attention to now,” Peter Slattery, an MIT FutureTech research scientist and one of the study’s co-authors, told MIT Sloan School of Management. 

BizTech reached out to MIT FutureTech Director Neil Thompson, who is also a principal research scientist at MIT CSAIL and IDE, to ask how leaders at financial organizations can respond to these concerns, which have been made more accessible at airisk.mit.edu/priorities. 

BIZTECH: What would you say are the biggest misconceptions in finance about AI risk? What are industry leaders missing? What do you think they're starting to understand better? 

THOMPSON: The biggest gap I see is an assumption that AI risks fit neatly into existing operational risk and control taxonomies. Our AI Risk Repository and prioritization work show several categories that do not. Some are about business and strategy rather than technology, such as power centralization, where a small number of actors control the development of AI, and competitive dynamics, where the rush to adopt AI leads to cut corners, inadequate safeguards and incidents. 

Companies are beginning to recognize the risks that fit their regulatory and risk management frameworks, such as model risk, privacy and cybersecurity. We do not yet have sufficiently detailed data to say whether the controls being put in place are adequate, especially for the most capable agentic systems now widely deployed. We have work coming out soon on how companies publicly disclose AI risk. 

Bigger picture and more speculatively, we also need to consider that AI, as a general-purpose technology, is likely to cause and exacerbate wider societal shocks and crises, meaning that leaders need to think beyond their operational risks and beyond their industry.

Click the banner below to learn more about continuous threat exposure management.

BIZTECH: Based on your conversations with other leaders in finance, what are the most concerning gaps in current incident response plans as AI risks sharpen? What can organizations handle within the sector immediately, and what would require more involvement from regulators in the long term? 

THOMPSON: We see that many operational risks of the generative AI era are already being widely discussed, to the extent that they fit within existing risk and control taxonomies. Regulated financial institutions in many countries are already required to maintain strong defenses against cybersecurity threats, fraud and scams, unjust discrimination and model risk. We do not have data on this, but our impression is that there is a significant response to those risks. 

The gap, as far as I can see, is in responding to risks from increasingly agentic deployments, and the way risk exposure radically changes between internal deployments, deployments between companies and deployments on the open web. There’s a great recent report from the Gradient Institute in Australia. 

BIZTECH: How should finance sector IT leaders think about meeting AI performance requirements without inadvertently expanding the attack surface in the process? 

THOMPSON: We are currently working on creating a map of mitigations for AI use across organizations, with the intent to start prioritizing these mitigations for priority risks by early 2027, but have nothing public to share at this time.