Skip to content

Diagnosing Bias

Tools for building healthcare AI governance into government procurement

Healthcare Procurement Guide:

How to write healthcare AI RFPs designed for privacy, bias prevention, intellectual property, technical transparency.

Table of Contents

*Any link with an arrow to its left like this can be expanded for further details and references.

Humaaans - 2 Characters.png

How to use this Toolkit:

The goal is for this kit to provide resources and knowledge on health technology procurement language and risks. By going through the following steps, you will end up with contract language to place into RFP documents.

Take some time to read through the
resources to get up to speed on health data and AI.
Use the
and walk through the steps in the app to generate the document language templates for your RFP.
Answer the customization questions in order to tailor the procurement templates for your particular use case. Use the
to find the right fairness metric for evaluating the performance of your procured model.
Build in the customized document language into your RFP
and publish the bid in a public-facing website for vendor bids and public comment!


To safeguard patient privacy and equitable access to high-quality care,
healthcare institutions should approach modern algorithmic tools with new oversight frameworks that build in transparency, fairness, and privacy by design.

Healthcare has historically held itself to a high standard when it comes to delivering high-quality treatment that prioritizes patient safety and privacy. The Health Information Portability and Accountability Act (HIPAA) has successfully promoted a culture of patient privacy protection for providers and the FDA has built a trusted institution for safely regulating innovative medical techniques and drugs.

The healthcare industry has a unique position to build good governance for technology and algorithms that impact patients. The range of algorithms used in healthcare is constantly expanding, encompassing diagnostic tools, risk stratification scoring, clinical note-taking automation, medical treatment recommendations, and resource allocation decision-making. This also opens up patients to a widening risk surface for subpar treatment outcomes, biased algorithm recommendations, and non-transparent decisions that lack logical and comprehensible explanations.

Notable research and journalism articles from the past few years have exposed the deep impact that healthcare algorithms can have on patient care: the racial gaps in a widely used Optum algorithm for
for kidney disease diagnosis, new sources of patient data collected by data brokers, uncovered by HIPAA, enable social determinants of health and lifestyle factors to
, and contributes to
for patients. There is a pressing responsibility for healthcare providers and algorithm operators to proactively define and require oversight of these algorithmic tools as they gain wider use and adoption.

Beyond potential bias within healthcare AI tools, there also remains continuous monitoring and evaluation concerns. Engineers from John Snow Labs
that a predictive readmission model that was trained, optimized and deployed at a hospital would start sharply degrading—and predicting poorly—within two to three months. A similar case occurred with IBM Watson’s oncology treatment recommendation service, where it was
after it began recommending unsafe treatments after experiencing model degradation.

This toolkit includes a set of educational resources for understanding healthcare AI products, provides a tool that helps generate procurement RFP language that hold AI companies accountable for their product outcomes, and outlines strategies in which procurement organizations can monitor and evaluate the performance of healthcare AI products over time.

The resources in this toolkit are iterative and constantly evolving — we welcome contributions and feedback to this work. Feel free to reach out to us at

Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
) instead.