If I Took It Out Today, Would an Outcome Change? | The 229 Podcast with Diane Constantine, Cathy Patterson, and Vicki Sergent

This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

If I Took It Out Today, Would an Outcome Change? | The 229 Podcast with Diane Constantine, Cathy Patterson, and Vicki Sergent

[00:00:06] Sarah: I always say hi to our production team. It's either to me, Landon or Holly, simply because they get to listen to this after and learn all kinds of neat stuff. So thank you, production team for creating this amazing group of women to have a fantastic conversation today. And if you're ready to get started, then I will go ahead and jump in. Are we ready? Yeah. Okay. Let's do it. Here we go. If an innovation saves clinicians time, prevents harm, and improves a patient experience, but cannot produce a clean 12-month ROI.
[00:00:13] Sarah: Why, does it deserve to scale? That is the decision health systems are being asked to make every day. And too often, we evaluate innovation with a financial lens that is important, but also incomplete. Today, we're moving beyond the question, did the pilot work? We're asking a more consequence. See, here we go. We are asking a more consequential question. What evidence is strong enough to scale it, stop it, or keep learning? I'm joined by three healthcare informatics leaders who make these decisions in complex environments.
[00:00:40] Sarah: Diana, Kathy, and Vicki, thank you for being here. And could you spend a few moments to introduce yourselves?
[00:00:48] Sarah: Sure.
[00:00:50] Sarah: Diane, you go first. Okay. Thank you. I'm Diane Constantine. I'm the Senior Director of Enterprise Health
[00:00:57] Speaker: Informatics at the Children's Hospital, Philadelphia. Welcome, Kathy. Hi, I'm Kathy Patterson. I'm the Chief Nursing Informatics Officer at Banner Health in Phoenix, Arizona. Excellent. And Vicki. Vicki Surgeon. I'm the chief nursing informatics officer for Fratered Thetacare Health in the South region.
[00:01:17] Speaker: This is a follow-up from our time together in NAPA where we had the first CNIO summit. It was such a raving success. We are almost full for next year and we have two of them listed. So I'm hopeful we keep growing this community. And thank you for being the first podcast as an output from that summit. I want to begin with a practical question at the center of this episode. And what is the framework you use to decide whether an innovation scale
[00:01:40] Speaker: stalls, stalls, or gets cut. We'll hear from each of you. And I'd love for you to name the framework and also give us the dimensions that really matter when you consider these scorecards. Diane, we'll start with you. What is your framework and where do patient safety, clinical usability and your operational value appear in it? Yeah. So we actually don't have a named
[00:02:01] Speaker: framework, but we use a multidimitional framework where financial ROI is part of it, but it's only one of the dimensions. We look across clinical and safety, like did we improve quality, reliability, safety? Also, patient, family, and workforce experience. You know, did we improve
[00:02:21] Speaker: the experience? Did we reduce burden? And then operational value. Did we improve workflow, efficiency, capacity, throughput? Ultimately, that that scorecard helps us to decide if we want to scale it, sustain it, maybe modify it, right? Or stop investing in it. And every
[00:02:43] Speaker: one of those dimensions doesn't have to be positive to justify a decision. A safety intervention may warrant scaling without traditional financial ROI, while something that saves money may not warrant scaling because it creates unacceptable clinical burden. So the scorecard isn't there
[00:03:03] Speaker: just to prove value, but to help us to decide what to do next. Yeah, I love that. Kathy, from your transformation, predictive intelligence workforce, operational lens, what framework do you use? We're pretty similar with Diane. We have a, we call it the nursing informatics intake form,
[00:03:25] Speaker: just because that's what the organization knows to request to get innovations show design and innovation. But it's really is a value realization card similar to the same domains of, is it safe? Does it provide quality? Does it reduce nursing burden? Here at Banner,
[00:03:46] Speaker: we have a patient persona. Her name is Sophia, and we really put Sophia in the middle of any of our value conversations or innovations. So first, what does it mean to our patient persona? and her family. And then reaching out, we think about how does that reduce the burden on our
[00:04:08] Speaker: clinicians and create that safe quality environment. Burden has been something we consistently talk about and have a strategic imperative for. So we're really releasing the time away from the keyboard and getting it back to Sophia as weighted significantly, probably more heavier than the
[00:04:30] Speaker: financial conversation. I like the persona aspect for sure. Thank you. And Vicki, complete the picture for us. What framework do you use and how does it connect frontline clinical impact to an enterprise decision? Yeah. So the framework we use here is our nursing technology strategy and prioritization matrix. So our elevator kind of statement for our nursing technology
[00:04:55] Speaker: strategy is simplify the work to amplify the care. But it really starts with a simple question. Does this technology strengthen the support around the nurse so they have more time and capacity for patients? We measure what matters at the front line. So documentation time, reduction, time and direct patient care, nurse and patient experience, communication responsiveness has
[00:05:19] Speaker: been a big one recently, and adoption of the new care models. And we pair these outcomes with direct feedback from nurses about whether the technology actually improved their workflow. So that creates that connection to enterprise decision making. We can move beyond asking whether a tool was successfully implemented and really determine whether it produced meaningful clinical
[00:05:39] Speaker: value. If it improves the nurse's environment, strengthens the human connection, and delivers measurable outcomes, we have that clear case to sustain, scale, and invest more in that technological solution. Yeah, so I'm hearing and what I'm loving, the scorecard isn't simply a list of metrics. It's really a decision model. Your dimensions.
[00:06:00] Speaker: are explicit, tradeoffs are visible, and the leaders know what evidence will change the
[00:06:05] Speaker: decision, even inclusive of that persona-based aspect of it. So great stuff. Thank you. One of the
[00:06:44] Speaker: easiest mistakes is to implement first and then decide what success means later. I want to focus on the measures and thresholds that are important to exist before implementation. So, Kathy, what clinical, workforce, operational, financial measures? Do you define
[00:07:01] Speaker: up front. So the team cannot move the goalpost after the launch. And if they do, then what's required of them to be willing to change some of those desired metrics. Yeah, a very loaded question and
[00:07:17] Speaker: thank you. Again, you know, for us it goes back to patient safety and quality. First and foremost, whatever innovation we bring from that operational or clinical perspective. does it have a direct positive impact on the patient in the bed?
[00:07:37] Speaker: And that is where it gets a little gray on how you go measure some of that because you do have the, what's the patient experience that comes after the age caps? Or if we think about from freeing our clinicians from burdensome device and, you know, device integration, if they're doing that manually entering data, when we free up their time,
[00:08:01] Speaker: it's sometimes hard to have a direct measurement. Typically, we'll see it with better patient outcomes. If I have more time to care at the bedside, my patient is more educated. They're in a safer environment. I'm looking at the bedrails. I'm looking at dependent loops and not looking at a keyboard.
[00:08:23] Speaker: So we always do measure the safety metrics first, depending how that relationship,
[00:08:34] Speaker: So one example we have is we had an initiative for ready to rescue where we were creating an innovation to get our SWAT nurses as surveillance capability. That was something from a workforce perspective where we wanted to free up our SWAT nurses
[00:08:52] Speaker: from using a paper-heavy process to review patients that were in the hospital for watchers, and we were able to measure it by reduced mortality. We could look at reduction of code blues outside of the ICU.
[00:09:11] Speaker: you. So in some instances, we do have those hard metrics, and other times it's knowing that we're there for the bedside clinicians and creating safe environments that is sometimes not that easily to measure. Yeah, I bet. Some of the variability and having to adapt it to what
[00:09:32] Speaker: the intent was for sure. So, Vicki, how do you account for workflow variation across sites, units or roles before comparing the results. And do you set scale or stop thresholds before you go live? Great question. And definitely not an easy one either. We do have multiple different
[00:09:54] Speaker: types of facilities within our organization. We're an academic medical center, but we have community sites and critical access sites as well. So, of course, everybody knows the workflows range quite a bit when you have those different environments. But we do account for variation by testing innovation in both our academic and community hospitals. We do have innovation units, which help us understand
[00:10:19] Speaker: how a solution performs across different regions, workflows, patient populations, and team structures before we make an enterprise decision. And metrics are really key as a part of this, but they only capture that one piece of the puzzle, which is the what. So we do take a really extensive, grounding and in-person approach with staff and seeing the technology in action to help us
[00:10:44] Speaker: understand that why. The why is the on-site support is essential during those innovation pilots and also the early implementations. So our success measures are defined before go live, but our scale or stop decisions combine the outcomes that we were measuring with adoption, as well as that frontline experience, workflow fit, and whether the solution can be supported.
[00:11:07] Speaker: reliably across our system. Yeah, I get a sense that you're both sharing that the metric might tell us what happened, but the thresholds that we either agree on or adapt help us decide what we need to do next, which is an important, I guess, distinction in many of these cases. I want to tackle the value that doesn't always land neatly on a P&L, and that's clinician time returned, that may be the
[00:11:29] Speaker: cognitive burden reduced, harm avoided, improvement of trust, or even the capacity that it creates. and how do you make some of those benefits credible through influence and even some of the executive decision-making? So a couple of perspectives I want us to tackle here. Diane, in a pediatric
[00:11:55] Speaker: rare but also consequential? Yeah. So I think this is where. relying exclusively on traditional ROI kind of gives us an incomplete picture. In pediatric, some of our highest value outcomes may involve events that are rare but potentially catastrophic. If an innovation reduces the likelihood of serious patient harm, we shouldn't have to manufacture
[00:12:19] Speaker: a dollar value to establish its importance. I don't think, I would say that doesn't mean we don't measure it. But not everything valuable has to be monetized, but I think it does have to be measured. And that measure could be in reduction of risk, greater reliability, reducing variation.
[00:12:41] Speaker: And in pediatrics, our patient experience frequently extends to the entire family. So if a technology reduces friction for a parent of a medically complex child or improves communication or makes care easier to navigate, that's meaningful, even if it doesn't appear on a P&L, right? And I think
[00:13:04] Speaker: the same applies to clinician burden and time saved. I would be cautious saying that every minute equals financial savings. You know, time becomes organizational value when we can demonstrate that we're able to use it a little bit differently. And I think we need to make the tradeoffs visible
[00:13:25] Speaker: enough that leaders can decide whether the evidence supports scaling, sustaining, stopping. You know, sometimes the value isn't the minutes we saved, but what the organization can now do with them. And what I appreciate most about rooms of being either CMIos or C&IOs is there's
[00:13:45] Speaker: a whole category. I feel like that needs to be created called clinician intuition. There's every once in a while that space where I hear because we know it's the right thing to do, or there's no way to measure this, but we know it's important to the patient or to the entire family. And the passion by which y'all show up to have that conversation in a boardroom
[00:14:05] Speaker: is unparalleled. And so I'm grateful that those are unmeasurable metrics that sometimes can have some of the biggest impacts. Kathy, when does the clinician time saved become true capacity or organizational value rather than a theoretical calculation?
[00:14:25] Speaker: Yeah, the clinical time saved, again, if we go back and think about removing the clinician from the keyboard and back to the bedside, isn't always a direct correlation with clinical outcomes. However, we do know there are studies that show when a nurse is able to spend more time
[00:14:49] Speaker: at the bedside, we do see quality increased. We see patient satisfaction increase. So as informatics and leaders, that's where we have to really draw the line of saying some of these innovations that we do to reduce the burden,
[00:15:09] Speaker: what we've seen in prolonged patient and safety outcomes and patient experience without being able to trace the five minutes saved here is going to equal an increase in this performance. So overall, I think that's back to your statement, Sarah, is we know getting
[00:15:30] Speaker: clinicians back to the bedside does provide value. And we see it in our, we call them voice surveys when we have nurses respond about their clinical environment on an annual basis. They will respond in that voice survey to say this innovation was able to give me more time in my,
[00:15:53] Speaker: space and breaks. You know, we can't ever forget about getting those nurses well-being to their lunches, getting their breaks, and reducing their cognitive burden for their overall wellness. It's like when someone in meeting ends early or somebody even canceled something on your calendar and you're like, oh my gosh, I have time back. And just as this joy of like room to feel like
[00:16:17] Speaker: you got through your day without having it follow you home in some cases is so key. Thank you. for sharing that. Many organizations have strong implementation ownership and a much weaker benefit realization ownership. In your organizations, once a project goes live, who owns the outcome and who has authority to intervene if the benefit is not materializing?
[00:16:40] Speaker: And so, Vicki, is it the operational leader, the clinical owner, IT, finance, the partner vendor, or is it a shared governance group that's taking a look at that in your organization? Yeah, this one's not an easy one either. Of course, the right answer is shared.
[00:17:00] Speaker: And for us at our organization, accountability is shared. But the key is it can't be vague. You know, each of those partners that you listed out has to own a distinct part of the outcomes. So for us, the operational leader is accountable for adoption and sustained workflow changes. The clinical owner ensures that the technological solution improves care,
[00:17:23] Speaker: supports that frontline practice. And then the technology team is responsible for reliability, integration, usability, usability, and support. We also do hold our vendors accountable to remain with us for performance and partnership, not just simply that implementation. We do call our
[00:17:45] Speaker: vendors partners. So it is something that we really rely on that partnership and integration with our vendors to be a true partner when we do these implementations on ongoing support. Additionally, you did bring up shared governance, and we do have shared governance groups that bring these perspectives together to review the outcomes, risks, and frontline feedback,
[00:18:07] Speaker: and then make decisions about whether to adjust scale or stop those innovation efforts. And ideally, shared governance does not mean shared ambiguity. That's always a little bit challenging, and I will say we're still struggling with that as an organization. And I think that's something as we mature more in that chair governance structure that I would like to see us become more and more refined on.
[00:18:32] Speaker: But we do name an accountable leader, clarify ownership before Go Live, and then remain collectively responsible for delivering that clinical and operational value we promised as part of the initiative. I was going to ask you, do you still have an executive owner? Because if everyone's responsible, no one is. And it just becomes this cycle of let's talk about it, but not make a decision. And then the alligator arms of no one's, you know, no one's responsible for it.
[00:18:55] Speaker: And the accountability isn't about blame, just about ownership of how something's happening inside of the org. So I'm glad to hear you say that. Diane, what should informatics continue to own after implementation? But then what needs to transition to operations? And what does that look like? I think about it even like an annual budgeting review process. Like where does it transition and how do you measure and look at the long-term effect? Yeah.
[00:19:19] Speaker: So I think in health care, we have. traditionally been very good at assigning accountability when we're implementing something. I think we're less consistent about assigning accountability for value realization. My view is that operations ultimately owns the outcome. If we're trying to improve safety, throughput, clinician experience, or revenue performance,
[00:19:41] Speaker: that outcome cannot belong to IT or informatics. But I don't think that means that informatics should disappear after the go live. I think informatics continues to help connect technology, workflow, and leading to outcome. And informatics should always be asking the questions, right? Like, you know, are we people actually using it?
[00:20:05] Speaker: Are they using it as intended? Did they establish a work around, you know, are there unintended consequences? And I think that the value isn't materializing, informatics can help determine why, you know, that is.
[00:20:22] Speaker: you know, I think that's why, that's what should inform really the next decision, the scale, sustain, modify, stop. I think operations owns the outcome. Informatics can continue to help the organization ensure that, you know, the technology is contributing to that outcome. So I guess in the end, I go live isn't the end of the informatics responsibility.
[00:20:46] Speaker: I think that's when we really need to start looking for that value realization. I would believe, too, that the informatics teams are always looking to see, is this still the right solution for us down a longer road? Because if it gets put into production, it has to be maintained, paid for, et cetera. You may say five years later, it's time to revisit if this is the right option for us as well.
[00:21:07] Speaker: Thank you. Adoption is often the first number everybody sees, but high usage can reflect genuine value. And I think about like a mandatory workflow or even curiosity. So when is adoption a meaningful, signal, when is it vanity? I want to grab two perspectives on this one. Back to you, Vicki, about what do you look for beyond logins and utilization? Is it sustained use,
[00:21:31] Speaker: the workflow completion, again, outcome improvement, or is there more to the equation?
[00:21:38] Speaker: I like what you said there, Sarah, about it being vanity. I think I might take that back with me as part of this podcast. Because you're right, you know, logins and utilization tell us, you know, solely whether somebody opened the technology or interacted with it, not whether it created value. So looking at that sustained adoption over time and whether the intended workflow
[00:22:01] Speaker: is actually being completed. And, you know, most importantly, are the staff creating, actually using the solution as intended, or are they creating workarounds to circumvent it? Then we connect the adoption to the outcomes of technology we intended to improve. Did it actually reduce the documentation burden? it gives the nurses more time with the patients? Did it improve communication, safety,
[00:22:25] Speaker: or the nurse and patient experience? So, you know, it is just one piece of that pie that we have to look at holistically when we say, did this innovation effort or this technology effort do what we thought it was going to do? We also have taken the perspective of looking at hosting post-go live
[00:22:50] Speaker: product. So really bringing the unit together with the frontline teams getting that feedback because as we mentioned a little bit before, that data shows us what's happening, but that getting that staff feedback and really understanding why it's happening is so valuable to these endeavors. And ultimately, success is not simply that a tool is being used. It's the functionality
[00:23:12] Speaker: has become a reliable part of their workflows. It actually improves their work and is producing meaningful clinical and operational outcomes that can be sustained. So I think having that adoption piece coupled really closely with that, did we bring the value we say we were going to bring with
[00:23:31] Speaker: this is crucial. Yes. And Kathy, when can low adoption mean the solution is wrong? And it's not that the users failed to make the necessary changes. Yeah, I think you have to look at it in a timeline perspective. Low adoption when you first implement doesn't necessarily mean it was a bad
[00:23:52] Speaker: solution. It could have been a bad implementation. And that could be, did we get the right education out? Did we get the right at the elbow support? So it's always the less vain moment, Sarah, that we go do a cycle of improvement and look at our own operations to make sure we've provided
[00:24:14] Speaker: the right type of innovation. And then obviously we always have to take in the adoption curve. You know, looking at a adoption, such as ambient listening, I think we have to understand you're going to have early adopters and then you're going to have late adopters. It doesn't mean that
[00:24:36] Speaker: the clinical quality and outcome you're looking for is a direct measure of the effectiveness of the innovation. So again, that's another, it's a little bit gray on the adoption.
[00:24:56] Speaker: Now, if the solution has been in the environment over time and you've done cycles of improvement, you've done education and it's not getting adopted. That is obviously our red beacon, our flag to identify that this isn't providing the value we wanted. And that can come in, not seeing the
[00:25:17] Speaker: outcomes, not seeing the user sign-ons. So it is an art of making sure that there's a timeline, the adoption curve, and then really always readdressing what you put into the environment for effectiveness. Well, let's be honest. It's not like everyone's waiting around for us to give them new tools all the time.
[00:25:37] Speaker: Even if it's exactly what everybody wanted, you still have to figure out how to incorporate it into an incredibly, not just busy, but like, you know, people's lives at stake in some of these solutions. And so even if it's something you want, it just takes time to get used to it. I mean, anytime we add anything to our routine, it just takes time. And I think there's a little bit of a grace that goes with
[00:25:58] Speaker: especially if it's the thing that people still want. All right. We're coming to almost a close. us here. I want to make this a rapid round. Tell us one scorecard decision that you're really proud of, something you either chose to scale, redesigned, or something that you said, we're going to stop on this one, whether the evidence took the decision or one thing that another system should be
[00:26:18] Speaker: copying and thinking about. Vicki, let's start with you. Yeah, thanks. To start, I just want to say proud that we have a framework to utilize. I think that's something that's not a standard across the industry. And I don't want to downplay the work and effort it took to create a framework and have buy-in across
[00:26:37] Speaker: our very matrix nursing leadership structure. And the fact that this framework was created in conjunction with our nursing executive leadership and our strategic planning is a huge win to bring our nursing leadership and technology solutions together to create lasting outcomes and to really help our IT-CI groups to know what is important by our nursing customers to focus on. And in the last
[00:27:01] Speaker: couple seconds. I do have a proud moment, so I want to share it. I am most proud of our, we just recently had a successful MyChart Bedside implementation. We started in our innovation units with the goal of scaling to the whole organization over a year with multiple phased rollouts. But based on the success within the innovation units and the adoption and feedback by the nurses in our innovation units, we were
[00:27:24] Speaker: actually up as a system with my chart bedside in three months with only one additional large big bang go live, which was absolutely phenomenal. So I just really want to say that having innovation units is a really great thing too. Hearing the voice from their nursing colleagues is something that really means a lot to their peers. So being able to have a group that we can test
[00:27:49] Speaker: these things out with and give that strong peer vote of confidence to. It's been really, really successful. Congratulations. Thank you. Kathy, what's your example?
[00:28:02] Speaker: We're really proud of a project we called Mission Possible, where we did safety check sheets. I think in the informatics space, we tend to have a lot of data-rich environments and sometimes they acknowledge poor or still have a lot of cognitive burden out at the bedside
[00:28:23] Speaker: because of the disparity of where all of the data relies and sits in the EMRs. So we created a nursing dashboard that we call our nursing care hub. And it really brings in length of stay throughput.
[00:28:40] Speaker: So all your operational metrics, it brings in patient experience from preferred languages to that ready to rescue work. We brought forward early warning scores and very visually intuitive graphing.
[00:29:00] Speaker: to the vital signs. And then we brought all the car harms. So that we always know, are we doing the right quality care for pressure injury reduction for our Clabcy, Cotty? And then peppered it with all those little pebbles and nurses are like, I have to go two clicks in
[00:29:19] Speaker: and go find this. So that's been a really great innovation. That's been adopted really great by our nurses. And then how I always measure, did we do well? I just simply go ask if I took that out today, either with outcomes change or your perception or cognitive load would return. And that is
[00:29:43] Speaker: one that the nurses will absolutely not let me take back away. And I think that's always a testament to giving them something of value. Oh, I like that. What if I took it away from you at this point? and see the reaction. Amazing. Thank you. Diane, bring us out. Okay. One decision I'm particularly proud of is we're changing the way that we're approaching our EHR optimization
[00:30:08] Speaker: and utilization. Instead of continually asking what else can we build, we're intentionally looking at the capabilities we've already implemented and asking whether we're realizing value from those. And so through some of our utilization work, we're looking at is the capability, configured and available? Are people using it? Is it addressing something meaningful?
[00:30:33] Speaker: And I guess most importantly, does greater utilization actually improve the outcome that we care about? I think that gives us a much more intentional decision, whether to scale, you know, we'll scale when we see that we're seeing meaningful value and we'll sustain if we're delivering at the right, you know, at the right level. And then we'll stop putting additional resources.
[00:30:56] Speaker: to something when the evidence just not, you know, furthering our investment. And I think the one thing I would tell another health system to copy is to look backward before you build forward. And not just with your EHR, but across your tech stack, because there may be tremendous unrealized values sitting inside the technology you
[00:31:20] Speaker: already purchase. Yes, going back and digging through to make sure it makes sense. And just thank you just for that perspective in general. Like there are things that throwing more resources at it is not going to make it better or fix it for sure. I'm going to close with one sentence from each of you, about 10 seconds each each.
[00:31:39] Speaker: And that is, what is the one question every executive team should add to its innovation scorecard tomorrow?
[00:31:48] Speaker: Kathy? I'm going to go back with the answer. If I took it out today, it would an outcome change. Awesome. Diane? What are? outcome are we trying to change and what evidence would tell us to scale, sustain, or stop? Perfect. And Vicki. Was our training, on-site support, and change management plan for frontline
[00:32:09] Speaker: teams about this innovation successful or are there things we have to adjust before we scale? So if you're listening, all three of those need to go into part of your due diligence perspective. Thank you. My takeaway is that measuring value on ROI does not mean lowering the bar or making the business case less rigorous. It means taking, it means making the full value
[00:32:33] Speaker: visible, financial performance, safety, quality, the capacity, the capacity, the clinician, patient experience, adoption, the fit, all of those different pieces and agreeing in advance how those signals will drive a decision. The best scorecard does not simply prove that a pilot was interesting. It gives leaders clarity and courage to scale what matters, redesign what can
[00:32:52] Sarah: improve, and stop what does not deliver. Diane, Kathy, Vicki, thank you for sharing the framework behind your decisions and congratulations on all of your success. Thank you. Thank you. Thanks. And for our audience, thanks for listening. That's all for now.