In healthcare, trustworthy AI starts with people

Bill Eggers: From Deloitte, this is Government’s Future Frontiers, the podcast that asks questions today to help create tomorrow. I’m Bill Eggers, executive director of Deloitte’s Center for Government Insights.

This episode was recorded in Amsterdam, at HLTH Europe, where professionals at the center of health innovation come together to discuss improving health outcomes for everyone.

Our guest is Dr. Anjali Bhagra. She is the medical director of enterprise and belonging at the Mayo Clinic. She sat down with my Deloitte colleague David Rabinowitz.

Here’s the conversation.

David Rabinowitz: Hello and welcome. We are here today to talk about trustworthy AI in healthcare and how we turn promise into progress. I’m David Rabinowitz, a life sciences and healthcare partner at Deloitte.

A lot of my work starts with the question of what do people in communities actually need to be healthy? And we can’t seem to have any conversation in healthcare these days without a heavy dose of innovation and AI.

We know that AI and innovation in healthcare won’t scale on the promise alone—but only when patients, when clinicians, when health systems really trust how it’s designed, how it’s governed, and how it’s used, and when each of those audiences really see and feel what matters to them.

For patients, that might mean feeling seen, respected, and protected. For clinicians, it’s about safety and reliability and does it fit into the workflow? And for health systems, it’s really about value—clinical value, operational value, and business value.

So, our focus for the podcast and conversation today is how do we earn the right for AI to matter in healthcare at scale and really be trusted. I’m thrilled to be joined today by Dr. Anjali Bagra. Anjali is the physician lead and chair of enterprise automation at the Mayo Clinic, and also the medical director for belonging. And I think that confluence of worldviews is going to be an important theme throughout the dialogue.

Anjali, thanks for being here.

Dr. Anjali Bhagra: Thank you for having me. Looking forward to our conversation.

Rabinowitz: So, Anjali, there’s enormous excitement around AI and healthcare. You can kind of feel it in the room here at HLTH Europe. There’s urgency, but there’s also skepticism, and some fatigue. I think people see the potential, [but] they also see and are a little bit worried about the complexity.

We do a regular survey of trust in AI across sectors and one of the things that we’re finding is a lot of unevenness, especially in life sciences and healthcare today. Workers that have high trust in their employer are using AI regularly, and they’re using it with some meaningful impact. But sometimes, that’s uneven—sometimes it goes up, sometimes it goes down. And usage—it’s not quite reaching the scale that we all hope and expect.

And for patients and for members of health plans, it’s an even different story: Only about 40% say they’re using and then really engaging with the AI tools made available to them. So, I’m curious, from your vantage point, does that track with what you’re seeing?

Bhagra: Yeah, David, that’s a great place to start. As you were sharing the figures, it’s clear that AI is here. It’s not abstract, and it’s certainly being used within pockets that include documentation—other areas where administrative burden is heavily impacting both the clinician as well as allied health staff, in the background, imaging, office procedures, [and] a lot of different things.

So yes, AI is here, but what makes it more complex and what impacts somewhat uneven uptake and the numbers that you were quoting is that healthcare is deeply human. There’s a lot of variation in how healthcare is delivered: It’s complex, it’s regulated, and there’s a lot of potential of it going wrong.

Technical performance, alone, does not guarantee trust in a situation like this where humans are standing to get impacted by this. So, trust keeps surfacing up because AI is entering positions where clinical decisions could be impacted by AI.

It’s really entering points where patient experience is impacted by AI. It’s also entering moments where outcomes are impacted by AI. And, I think, fundamentally, there are deeper questions: What data was used in training the AI, for example? Does it work for everyone? Are there pockets being left out? What happens when it goes wrong? Who is liable? Who is accountable?

Rabinowitz: You bring up a lot of questions that I think a lot of organizations are starting to service and wrestle with, but sometimes, it feels like we treat […] and we’re tempted to deal with trust as a communications challenge and a question of if only we explain it in a better way, more often, [in a ] more accessible [manner], adoption will naturally follow and people will kind of come on the journey with us.

We know that trust is really key and really power really paramount and it’s got to be earned—earned through transparency, through safety, through consistency, and also earned by people seeing outcomes that matter to them. Not to mention the continued need to deal with longstanding mistrust and distrust in institutions. So, how do you think about the need to go beyond the communication that you’re talking about and really start to show the value?

Bhagra: So, exactly, trust is not a slogan. It really is built through repeated evidence that the technology is working, and is driving the outcome it was designed to; that it’s not harming humans, and then, there’s accountability at the center of AI.

And that matters because AI is not entering a blank state: There’s history. There’s history of marginalization. Some communities have experienced harm more than others. Some patients have candidly felt unheard, and some clinicians on the supply side feel burdened by this technology and don't always see a clear value where it’s helping them in their day-to-day work. So, AI really enters a system where there’s history and there’s context.

Rabinowitz: I think it’s such an important point that all of this talk of adoption is happening amid not just trust gaps, but health gaps. You talked about communities that have been harmed, that have been excluded, that have been misrepresented and underserved. And at the same time, there’s this trend of more and more information coming out, patients coming […], consumers that we believe are empowered to take their own health journeys and their own health [in their own] hands.

At some point, it comes down to leadership. And I know that you’ve talked a lot about what leaders should be doing and how leadership really matters in recognizing that context. Talk about the leadership challenge in front of us.

Bhagra: Yeah, I really think it all boils down to what the leadership challenge is, and how leaders will navigate that challenge. We know that health systems alone cannot assume the responsibility of information and guidance, because it’s a fact that our patients are reaching out to many other sources for this, and if patients fundamentally do not trust the health system, they may turn elsewhere.

So, I think this is a really important pivotal point where leadership matters and [it is] the clarity, the transparency, the humility, as well as the ability to get in[to] our patients’ shoes by leaders [that] will determine the trust that we will build with the patients.

Rabinowitz: You know, I want to dig deeper into a couple of the audiences that we’ve been talking about. We’ve been talking about patients, we’ve been talking about clinicians, we’ve been talking about health systems. They all might want to have trust in these tools and technologies, but they might be asking different questions.

Let’s start with patients: You know, in your practice and in your roles, as you think about the role of AI with patients, what do they need to have trust in the technology?

Bhagra: Yeah, I have this question come across in many patient encounters, and I would say the no. 1 thing that I hear from my patients, [and] many other clinicians share [this] is if they will continue to be able to see their physicians. So, in other words, will AI replace their physicians?

Will they be heard? Will their stories just become data points? Will they be able to establish the trust and the relationship with the physicians? Can they move through a system with that level of trust?

So, fundamentally, I think it’s an existential question around the patient-physician relationship and what will happen to that. And I think that fundamentally is the question that the patients have.

Rabinowitz: Switching gears a little bit, let’s switch to clinicians. We know they’re often very supportive of innovation, but you know, [they’re also] appropriately cautious, given the responsibility they take on for patient care […] from a regulatory perspective […] also from a clinical workforce standpoint. What does trust mean from their vantage point?

Bhagra: Yeah, so [from my] firsthand experience as a clinician, I would say the most important thing from a clinician’s standpoint is what goes into building that AI. Can I trust this black box? Is it generating outputs that I can rely on? What is the amount of fact-checking that I need to do as a clinician?

So, I think it’s really important to have transparency and a good understanding of how the AI is designed, what outcomes it has been designed to drive, and does it really move the patient outcome and experience to the next level?

So, fundamentally, I think that’s really important. The other piece that comes along with that is this just one extra thing for me to do or is this really going to help lessen the burden of providing healthcare?

So, is it there to reduce the friction of healthcare delivery? Is it really helping me enhance access for my patients? Is it really helping me drive the outcomes for my patients and is it really helping drive the patient experience?

Rabinowitz: It’s a lot to put on a workforce that already has a lot on their plates, which, I guess, brings us to the perspective of the enterprise, the health system lens on this.

You know, leaders are balancing a tremendous amount right now—risk, quality, financial sustainability, workforce capacity, reputation, as well as a regulatory environment that continues to be quite complex.

From your health system leadership perspective and the work that you do at the enterprise level, what does trust look like at the organizational level?

Bhagra: I’d say, at a health system level, trust requires clarity around governance. Trust requires clarity around which tools are used, who approves them. How are they monitored? How are their performances measured? And ultimately, how are the risks escalated?

I think those are really important ingredients when it comes to a systems-level view. The other piece I would say to add onto that is what kind of debt does it incur to the organization versus what kind of capacity does it truly build for the organization—both from a patient perspective as well as a workforce perspective.

Rabinowitz: You shared a lot about the importance of governance and leadership and the processes that have to be in place at multiple levels. But I think about some of the other hats that you wear, [and] from a belonging perspective, how do you see culture playing a role in this conversation?

Bhagra: I think culture has the biggest role to play in any shift within healthcare. Fundamentally, building trust, building faith, and really moving an entire workforce along are integrally dependent on the culture. So, in my view, culture is the biggest bottleneck and the biggest facilitator of any capability within healthcare.

So, it plays a big role, and it’s complex. We can break it down into components like governance; good governance, transparent governance, help build a more permissive culture. The flipside of that is unclear governance, lack of transparency, lack of clear workflow benefits [which] lead to an erosion of the culture.

So, fundamentally, I see […] trust in AI to be heavily dependent on culture stewardship and as possibly the most important work of leadership, when it comes to building trust.

Rabinowitz: So let’s try and move that into some practicalities and some clear considerations for others to really think about. As you’re thinking and you’re working to really build trust and scale, what are some of the building blocks that leaders really need to focus on?

Bhagra: Yeah, I would first start with purpose; I think that’s fundamental. What problems are we really trying to solve? What’s the purpose here? I think that really defines if AI should be adopted, and how it should be adopted. From there, I think leaders need to work on a few other building blocks.

The second I think about is the data architecture and technology foundation. Because you can’t really build anything without that. Trustworthy AI truly depends on high-quality data that’s inclusive, fit-for-purpose tools, appropriate validation methodology within the organization, post-deployment monitoring, as well as infrastructure to understand where it’s creating the intended impact and where there is drift.

Third, I would say, is governance and accountability. Clarity around accountability and what role AI or any tool is playing within the organization is absolutely critical. And the final one I would say is partnership. Because you can’t do this alone, and no single organization, no single industry can do this alone. And partnership is what allows us to learn from each other and really build those blocks together.

Rabinowitz: Going a slight bit deeper on this notion of partnership and engagement, we often talk about the need to build with and not build for. But sometimes, that can be an easy thing to say and a hard thing to do in practice—especially in practice at scale.

How do you and how should leaders really think about both involving clinicians and also patients in ways that are meaningful in building with and not just symbolic?

Bhagra: Yeah, that’s a great question, David. I mean, I really like to think about AI as a means and not the end. And, really, those relationships allow us to help prioritize—prioritize the use cases where need is meaningful—both for the patients as well as clinicians where evidence is strong, where it’s compelling that an integration would bring the benefit; where workflow is ready, because you can’t bring in a technology where the workflow is messy and not clearly defined; and finally, where there is governance to help responsible deployment.

Rabinowitz: Talk a little bit more about how that plays out from a patient perspective. We know that, especially for communities that have experienced less than optimal healthcare for a variety of reasons, the level of trust that they might be coming into an experience [or] an encounter [with] might be different. So, from a patient or community perspective, what are you seeing as the ways to think about building with in that reality?

Bhagra: Yeah, another very critical component of the build: I would first say that patient engagement cannot and should not be an afterthought. So, in the building cycle, it’s critically important for patients to be a part and show what transparency should look like.

What is their fundamental understanding of the tool—the capability of the tool? Do they have any concerns about the data set that’s being used? Do they have the right level of confidence? And would they be comfortable with these tools being deployed in the clinical care environment? Would these tools really help enhance versus erode the patient-physician relationship? So, all of those things are critically important, and I would say the patient engagement should be the beginning point and not the ending point.

Rabinowitz: You know, given how complex the management systems have to be to accomplish everything that you’ve been talking about, I imagine that accountability is a really important question. And for many organizations, accountability for who really is accountable for this agenda can be fragmented.

Bhagra: Yeah.

Rabinowitz: And you’ve been talking a lot about thinking about this not just as a data challenge, as a technology challenge, as a workflow challenge, but really a holistic set of considerations. So, how should leaders be thinking about accountability as we bring these tools into the day to day?

Bhagra: So, I’d first say, healthcare is a team sport. When we think about accountability, I think we bring that same mindset in because AI can neither be developed nor deployed successfully in silos. What that means is we need a combined accountability model, which brings strategy, governance, risk, clinical operations, research—everybody—on the table.

So, it’s a fundamentally different way of doing things. It transcends the legacy departmental model when it comes to accountability. And so, it really behooves us as leaders to think about accountability more creatively, but also with a very strong lens of ethics and keeping in mind what are the voices that we are hearing? What are the voices we are drowning when it comes to build, but then also accountability?

Rabinowitz: It’s interesting to kind of add that to your set of building blocks and really think about organizational choices as part of a building block to the strategy here. And it almost turns trust in AI into a bit of a leadership discipline.

Let’s start to close with what to do about it: If I’m a leader in healthcare listening to this discussion, I certainly understand the importance of trust, understand the different perspectives from a patient, clinician, a health system standpoint, and a lot of different building blocks to put in place to be successful. But what kind of commitments should I be making right now?

Bhagra: Yeah, I would say the first one is humility. AI is undoubtedly very powerful, but we still have a lot to learn. So, I would say begin with humility.

Second is, be guided by evidence: This is not new in healthcare. Evidence-based practice has defined our learning and practice. And it’s absolutely essential to be clear about what’s driving our purpose and where we are creating real value.

The third, I would say, is transparency. It’s okay if things aren’t working and we ought to have the courage to decommission [them]. And for that, we need to be transparent in communicating, transparent in how we operate, transparent in what accountability we hold toward our stakeholders here, who are our patients.

And then, finally, I would say a fierce commitment to equity. I know we’ve been talking a lot about trust and context and history in healthcare, and without having a clear commitment to designing for all and not for some, I think we would be really missing true leadership in AI.

Rabinowitz: So, that’s a really thorough and thoughtful list. I just might add one more, which is around continuing to interrogate, continuing to ask better questions.

Bhagra: I’m optimistic about that, David. I think people in healthcare are asking the right questions, and we’ve got to keep going. Ask more questions, ask as many questions [as you need], and ask the right questions.

Clinicians truly want helpful tools, and our patients truly want a better experience. And as long as we are asking questions that address those two things, I think we’re on the right track.

Leaders do see AI’s potential, and we also see the need for more responsibility and combined accountability. We have to keep the human purpose at the center. We have to keep our patients at the center. The goal ultimately is better health and better capacity to make our patients healthier and happier.

Rabinowitz: Such a wonderful message, Anjali. Thank you for joining us, and thank you for a thoughtful and thorough conversation.

Bhagra: Thank you, David. I really enjoyed it.

Thanks for listening to this episode of Government’s Future Frontiers from Deloitte. You’ll find all our previous episodes wherever you get your podcasts. And to make sure you don’t miss new ones, be sure to follow the show on your favorite podcast platform.

This podcast is produced by Deloitte. The views and opinions expressed by podcast speakers and guests are solely their own and do not reflect the opinions of Deloitte. This podcast provides general information only and is not intended to constitute advice or services of any kind. For additional information about Deloitte, go to Deloitte.com/about.