Newsroom

Artificial intelligence is rapidly transforming many sectors of the economy. In the high-stakes environment of healthcare, the intersection of AI-driven analysis and human choices raises important questions. A recent study by Professor John Edward Joseph of the UCI Paul Merage School of Business and colleagues at UCI Health dives deep into these questions, revealing surprising insights about how AI can both elevate and challenge our decision-making processes—whether in diagnosing medical conditions or guiding business strategies.

Joseph and coauthors David A. Fussell, Cynthia C. Tang, Jake Sternhagen, Varun V. Marrey, Kelsey M. Roman, Jeremy Johnson, Michael J. Head, Hayden R. Troutt, Charles H. Li, Peter D. Chang, and Daniel S. Chow published “AI Efficacy as a Function of Trainee Interpreter Proficiency: Lessons from a Randomized Controlled Trial” in the American Journal of Neuroradiology.

Inspiration: AI and Medical Diagnostic Accuracy

Joseph’s study was rooted in the growing integration of AI into critical decision-making processes. “AI is becoming an important tool, not just for healthcare but for managers making daily strategic decisions,” he says. His interest in strategic decision-making, combined with ongoing collaborations with UCI Health, provided a natural extension to explore how AI could enhance diagnostic accuracy and efficiency in medical settings. “The healthcare side is ahead of the curve in AI adoption, which offered a unique context for us to examine these dynamics,” Joseph says.

The Study: AI and CT Scans for Stroke Diagnosis

To explore the effects of AI on diagnostic decision-making, Professor Joseph and his team conducted a randomized, controlled trial at UCI Health, focusing on the use of AI-assisted tools in the diagnosis of stroke using patient CT scans. “We took advantage of a tool employed at UCI Health, which was an AI designed to aid doctors in stroke diagnosis,” says Joseph. The study involved medical students and residents, who were asked to evaluate 200 CT scan images, half of which were manipulated by AI to read either as correct or incorrect.

The primary goal was to assess the AI’s effect on diagnostic accuracy and efficiency, particularly in terms of how experience levels influenced outcomes. Participants completed a pre-survey to gauge their biases toward AI, which was later correlated with their diagnostic performance. By comparing the performance of these trainee interpreters with and without AI assistance, the study sought to answer critical questions about AI’s role in enhancing or potentially hindering decision-making processes.

AI’s Efficacy Tied to User Experience and Bias

A significant finding from Professor Joseph’s research is that AI’s efficacy is closely tied to the experience level of the user. “Accuracy improves for everybody but especially for medical students,” he says. The study revealed that medical students saw a 11% improvement in accuracy when using AI, while residents experienced a more modest 5% increase which was not significant. This suggests AI serves as a more substantial aid for less experienced practitioners, helping them bridge the gap in diagnostic proficiency.

Additionally, the study found that AI not only helps to improve accuracy, but it also encourages users to think more critically, especially in cases of disagreement between AI and human judgment. “We found that AI can improve accuracy, but it also improves information processing more generally,” Joseph says. This heightened vigilance was particularly beneficial for avoiding false negatives, where AI-assisted diagnostics caught issues human interpreters might have overlooked.

The study also highlighted the role of bias in AI efficacy. “Participants who had a positive bias toward AI tended to perform better,” says Joseph, “while those with a negative bias demonstrated greater vigilance, which also led to improved outcomes. Negative bias actually creates more vigilance, which emphasizes the importance of maintaining a healthy skepticism.”

AI and Strategic Decision-Making in Business

While the study was conducted in a medical context, its findings have significant implications for the business world, particularly in strategic decision-making. Professor Joseph draws parallels between the use of AI in healthcare and its application in business scenarios, such as financial analysis, competitive strategy, and product development. “It’s no different from a CEO looking at AI output in the competitive environment or a board of directors evaluating strategic plans,” he says.

This research underscores the importance of understanding the context in which AI is deployed and recognizing that its benefits vary depending on the experience and biases of the users. “In healthcare, missing a diagnosis is critical, whereas in business, a false positive in launching a new product could result in significant financial losses,” Joseph says. This nuanced understanding of AI’s role can help managers make more informed decisions about when and how to integrate AI into their strategic processes.

Study Glimpses Future Implications for Harnessing AI Across Industries

Professor Joseph’s research offers valuable insights into the nuanced relationship between AI and human decision-making. “This study not only highlights the potential of AI to improve accuracy and efficiency in healthcare but also provides a broader framework for understanding its implications in the business world,” he says. As AI continues to evolve, these findings will undoubtedly contribute to shaping how organizations leverage this powerful tool in making strategic decisions that drive value.

Joseph’s future research plans have the potential to significantly advance our understanding of AI’s role in both education and business. By continuing to explore how AI can be most effectively integrated into various strategic decision-making processes in different industries, and by examining the long-term effects of AI use, ongoing research will provide broader insights that can guide organizations across industries to harness the full potential of AI.