Healthcare AI Procurement Guide

How to write healthcare AI RFPs designed for privacy, bias prevention, intellectual property, technical transparency.
Table of Contents

*Any link with an arrow to its left like this can be expanded for further details and references.
Just like this! Click on the original arrow to collapse this piece of text back inside.

Humaaans - 2 Characters.png
(Graphic courtesy of )

Introduction
To safeguard patient privacy and equitable access to high-quality care, healthcare institutions should use best practices in algorithmic governance to oversee procured healthcare technology, prioritizing transparency, fairness, and privacy by design.

The healthcare industry has historically held itself to a high standard when it comes to delivering high-quality treatment that prioritizes patient safety and privacy. The Health Information Portability and Accountability Act () has successfully promoted a culture of patient privacy and security protection in healthcare, and the (FDA) has evolved into a trusted institution for safely regulating innovative medical techniques and drugs.

The industry is in a unique position to establish strong standards of governance for technology and algorithms that impact patients. The range of algorithms used in healthcare is constantly expanding. Algorithms are now used in , , the , and in pharmaceuticals and disease modeling. Without proper precautions, such widespread use of algorithms risks exposing patients to subpar treatment outcomes, biased algorithm recommendations, and non-transparent decisions that lack logical and comprehensible explanations.

Recent trends around technology use in healthcare have exposed the negative consequences that healthcare algorithms can have on patient safety, privacy, and equity:

Racial biases in an algorithm made by Optum, a healthcare solutions company, that predicts resulted in a 28.8% gap of Black patients not being recognized for follow-up healthcare.
The use of in the diagnosis of kidney disease led to the under-diagnosis and delay of care for 29% of Black patients with advanced kidney disease.
New sources of behavioral and demographic data, such as social media history, internet searches, TV watching habits, and education status, are being used to .
Another healthcare software company, Epic Systems, developed an algorithm predicting sepsis risk and severity using flawed data that .

Healthcare providers and algorithm users need to proactively define and require oversight of algorithmic tools as they gain wider use and adoption.

These tools also require continuous monitoring and evaluation. A product from John Snow Labs that predicted emergency room readmission rates exhibited within 2 to 3 months of use. Similarly, IBM Watson’s oncology treatment recommendation service was after it began recommending unsafe treatments without adequate performance monitoring.

This toolkit is geared toward procurement officers in government agencies or public healthcare systems who are responsible for purchasing health technology. The site includes a tool that helps generate language for procurement requests for proposals (RFP) that holds AI companies accountable for their product outcomes. It also includes set of educational resources for procurement officers to better understand healthcare AI products, and a set of strategies for procurement organizations to monitor and evaluate the performance of healthcare AI products after they have been purchased.

The resources in this toolkit are constantly evolving. We welcome contributions and feedback to this work. Feel free to reach out to us at .

How to Use This Toolkit
The tool outlined in this paper aims to enable procurement officers in government agencies or healthcare systems to develop procurement contract language that takes into account the risk of algorithmic bias. The following steps are designed to help develop contract language that can be placed into requests for proposals or other documents.

Read through the to get up to speed on the principles that underlie the procurement language tool, and learn about additional recommendations for procurement officers.
Use the and walk through the steps in the app to generate the document language templates for plugging into your RFP.
Build the customized document language from Step 1 into your RFP and publish the bid in a public-facing website for vendor bids and public comment. The template also includes an evaluation framework that can be used to grade competing vendor bids.
Use the guiding questions within the at each stage of the model lifecycle to promote transparency and accountability throughout the design process.

Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.