Healthcare AI Procurement Guide

Procurement Language Generator

AI Model Lifecycle Checklist

Glossary

Explore

Healthcare AI Procurement Guide

⁠

How to write healthcare AI RFPs designed for privacy, bias prevention, intellectual property, technical transparency.

⁠

Design Principles for Healthcare AI 101⁠

⁠

Procurement Language Generator⁠

⁠

AI Model Lifecycle Checklist⁠

⁠

Glossary⁠

⁠

*Any link with an arrow to its left like this can be expanded for further details and references.

Just like this! Click on the original arrow to collapse this piece of text back inside.

⁠

⁠

(Graphic courtesy of

blush.design⁠

)

Introduction

To safeguard patient privacy and equitable access to high-quality care, healthcare institutions should use best practices in algorithmic governance to oversee procured healthcare technology, prioritizing transparency, fairness, and privacy by design.

The healthcare industry has historically held itself to a high standard when it comes to delivering high-quality treatment that prioritizes patient safety and privacy. The Health Information Portability and Accountability Act (

HIPAA⁠

) has successfully promoted a culture of patient privacy and security protection in healthcare, and the

Food and Drug Administration⁠

(FDA) has evolved into a trusted institution for safely regulating innovative medical techniques and drugs.

The industry is in a unique position to establish strong standards of governance for technology and algorithms that impact patients. The range of algorithms used in healthcare is constantly expanding. Algorithms are now used in

diagnostic tools⁠

risk stratification scoring⁠

, the

automation of clinical note-taking⁠

, and

innovative research⁠

in pharmaceuticals and disease modeling. Without proper precautions, such widespread use of algorithms risks exposing patients to subpar treatment outcomes, biased algorithm recommendations, and non-transparent decisions that lack logical and comprehensible explanations.

Recent trends around technology use in healthcare have exposed the negative consequences that healthcare algorithms can have on patient safety, privacy, and equity:

Racial biases in an algorithm made by Optum, a healthcare solutions company, that predicts

patient healthcare utilization and costs⁠

resulted in a 28.8% gap of Black patients not being recognized for follow-up healthcare.

The use of

race-based correction factors⁠

in the diagnosis of kidney disease led to the under-diagnosis and delay of care for 29% of Black patients with advanced kidney disease.

New sources of behavioral and demographic data, such as social media history, internet searches, TV watching habits, and education status, are being used to

set prices for insurance premiums⁠

Another healthcare software company, Epic Systems, developed an algorithm predicting sepsis risk and severity using flawed data that

led to inaccurate sepsis risk predictions for patients⁠

Healthcare providers and algorithm users need to proactively define and require oversight of algorithmic tools as they gain wider use and adoption.

These tools also require continuous monitoring and evaluation. A product from John Snow Labs that predicted emergency room readmission rates exhibited

severe performance degradation⁠

within 2 to 3 months of use. Similarly, IBM Watson’s oncology treatment recommendation service was

withdrawn from the market⁠

after it began recommending unsafe treatments without adequate performance monitoring.

This toolkit is geared toward procurement officers in government agencies or public healthcare systems who are responsible for purchasing health technology. The site includes a tool that helps generate language for procurement requests for proposals (RFP) that holds AI companies accountable for their product outcomes. It also includes set of educational resources for procurement officers to better understand healthcare AI products, and a set of strategies for procurement organizations to monitor and evaluate the performance of healthcare AI products after they have been purchased.

The resources in this toolkit are constantly evolving. We welcome contributions and feedback to this work. Feel free to reach out to us at

healthytechbias@gmail.com⁠

How to Use This Toolkit

The tool outlined in this paper aims to enable procurement officers in government agencies or healthcare systems to develop procurement contract language that takes into account the risk of algorithmic bias. The following steps are designed to help develop contract language that can be placed into requests for proposals or other documents.

Read through the

Design Principles for Healthcare AI 101⁠

to get up to speed on the principles that underlie the procurement language tool, and learn about additional recommendations for procurement officers.

Use the

Procurement Language Generator⁠

and walk through the steps in the app to generate the document language templates for plugging into your RFP.

Build the customized document language from Step 1 into your RFP and publish the bid in a public-facing website for vendor bids and public comment. The template also includes an evaluation framework that can be used to grade competing vendor bids.

Use the guiding questions within the

AI Model Lifecycle Checklist⁠

at each stage of the model lifecycle to promote transparency and accountability throughout the design process.

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.

Healthcare AI Procurement Guide

How to write healthcare AI RFPs designed for privacy, bias prevention, intellectual property, technical transparency.

Table of Contents

Introduction

How to Use This Toolkit