With AI, health IT leaders must link technology to purpose, CIO says

AI Health Technology Purpose: CIOs Must Link Tools to Patient Outcomes

Sarah Mills | GlobalBeat

Health IT chiefs must connect every AI deployment to a measurable clinical goal or risk wasting resources, Cleveland Clinic CIO Matt Kull told the HIMSS conference in Orlando.

Kull said his team rejects 7 of 10 AI pitches because vendors cannot prove the tool will improve care or cut costs.

The warning comes as U.S. hospitals spent $7.2 billion on AI software in 2023, a 25% jump from the prior year, according to the Healthcare Financial Management Association. Regulators have cleared more than 750 medical AI models, yet adoption stalls when clinicians see no clear benefit.

Kull, who oversees technology for Cleveland Clinic’s 22 hospitals, said he requires each proposal to show a 10% gain in efficiency or a 5% drop in adverse events within 12 months. “If the algorithm cannot meet that bar, we don’t pilot it,” he told reporters after his keynote. The clinic’s review board has approved 18 AI projects since 2022, including tools that predict sepsis and shorten MRI wait times.

Other systems told the meeting they use similar filters. Kaiser Permanente only funds pilots that save at least $500,000 or reduce readmissions by 2%, Associate CIO Sarah Azar said. Intermountain Health demands a peer-reviewed study before any roll-out, CTO Craig Richardville confirmed. The strict criteria have led Intermountain to cancel 4 radiology AI contracts this year, he added.

Clinicians back the tougher stance. Dr. Patricia Garcia of Chicago’s Lurie Children’s Hospital said she stopped using an AI scribe after it added 5 minutes to each visit. “The vendor promised faster notes, but the opposite happened,” Garcia said. She now insists on a 2-week ward test before signing any license.

Data quality worries also limit uptake. FDA officials told the conference that 12% of cleared algorithms failed when used on new patient populations. Deputy Director Dr. Suzanne Schwartz said hospitals often feed the models imaging files that differ from the training set, producing “unexpected and sometimes dangerous” outputs. She urged CIOs to demand site-specific validation studies.

Financial officers echoed the caution. Tricia Guidry, CFO of Ochsner Health, said her Louisiana system shelved an AI staffing tool when overtime costs rose 3%. “We budgeted for savings, but the predictions were off,” Guidry said. The nonprofit now caps AI spending at 1% of its annual $500 million capital budget until clearer returns emerge.

Start-ups are adjusting their pitches. Aidoc CEO Elad Walach said his firm now offers outcome-based contracts that charge only if hospitals hit agreed targets. The company signed 8 such deals in the first quarter, double the number in 2023, Walach told investors on a Tuesday call. Rival Viz.ai announced a similar model last month.

Academic centers see the shift as a chance to lead. Stanford University will launch a national AI outcomes registry this summer, Associate Dean Dr. Nigam Shah said. The database will track mortality, length of stay, and cost for every deployed algorithm, aiming to give hospitals real-world evidence within 18 months, he added. More than 50 systems have pledged data, including HCA and Mayo Clinic.

Regulators are watching the trend. Health and Human Services Secretary Xavier Becerra told attendees the agency will propose rules this fall requiring hospitals to report AI performance metrics. “Taxpayers deserve proof these tools work,” Becerra said. He gave no details on penalties for non-compliance.

Investors took notice. Shares of AI health firms fell 3.4% on Wednesday, the biggest drop since January, according to Bloomberg data. Analyst Jeff Garro said buyers want “proof, not promises” before renewing licenses.

Background

Hospital spending on artificial intelligence has tripled since 2020, driven by vendor claims that the software can ease staff shortages and improve care. Early hype focused on imaging tools, yet studies showed many products performed worse on minority patients, prompting FDA safety letters in 2022. CMS last year began requiring hospitals to list AI models used for Medicare billing, but did not demand outcomes data.

What’s Next

Kull said Cleveland Clinic will publish results from its 18 active AI pilots in a peer-reviewed journal by December, adding that systems which fail to meet targets will be shut down. Becerra confirmed HHS will hold a public hearing on proposed AI reporting rules in September, with final wording expected by March 2025.

The tighter scrutiny marks a turning point for an industry long on promise and short on proof, analysts said. Hospitals that cannot show AI works may face budget cuts as federal relief funds expire, Guidry noted. Vendors that deliver verified savings are likely to win the next wave of contracts, Walach predicted.