HealthyTechBias

Healthcare AI 101

Procurement Language Generator

Fairness Metric Quiz

Responsible AI Product Design

Glossary

Explore

Health Tech Equity

Nidhi Hebbar

Matthew Zhou

⁠

Healthcare Procurement Guide:

How to write healthcare AI RFPs and contracts designed for privacy, bias prevention, intellectual property, technical transparency.

⁠

Healthcare AI 101⁠

⁠

Procurement Language Generator⁠

⁠

Fairness Metric Quiz⁠

⁠

Responsible AI Product Design⁠

⁠

Glossary⁠

⁠

*Any link with an arrow to its left like this can be expanded for further details and references.

Just like this! Click on the original arrow to collapse this piece of text back inside.

⁠

⁠

How to use this Toolkit:

The goal is for this kit to provide resources and knowledge on health technology procurement language and risks — it should be as easy as possible to go through the steps here and end up with contract language to put into your RFP document.

It’s as easy as 1234...

Take some time to read through the

Healthcare AI 101⁠

resources to get up to speed on health data and AI.

Use the

Procurement Language Generator⁠

and walk through the steps in the app to generate the contract language templates for your RFP.

Discuss through the customization questions in order to tailor the templates for your particular use case. Use the

Fairness Metric Quiz⁠

to find the right fairness metric for evaluating the performance of your procured model.

Build in the customized contract language into your RFP and publish the bid in a public-facing website for vendor bids and public comment!

Introduction

To safeguard patient privacy and equitable access to high-quality care, healthcare institutions should approach modern algorithmic tools with new oversight frameworks that build in transparency, fairness, and privacy by design.

Healthcare has historically held itself to a high standard when it comes to delivering high-quality treatment that prioritizes patient safety and privacy. HIPAA has successfully promoted a culture of patient privacy protection for providers and the FDA has built a trusted institution for safely regulating innovative medical techniques and drugs.

The healthcare industry has a unique position to build good governance for technology and algorithms that impact patients. The range of algorithms used in healthcare is constantly expanding, encompassing diagnostic tools, risk stratification scoring, clinical note-taking automation, medical treatment recommendations, and resource allocation decision-making. This also opens up patients to a widening risk surface for subpar treatment outcomes, biased algorithm recommendations, and non-transparent decisions that lack logical and comprehensible explanations.

Notable research and journalism articles from the past few years have exposed the deep impact that healthcare algorithms can have on patient care: the racial gaps in a widely used Optum algorithm for
patient care utilization and costs⁠
,
race-based correction factors⁠
for kidney disease diagnosis, and new sources of patient data collected by data brokers, uncovered by HIPAA, enable social determinants of health and lifestyle factors to
impact insurance premiums⁠
, and contribute to
inaccurate predictions on sepsis risk and severity⁠
for patients. There is a pressing responsibility for healthcare providers and algorithm operators to proactively define and require oversight of these algorithmic tools as they gain wider use and adoption.

Beyond potential bias within healthcare AI tools, there also remains continuous monitoring and evaluation concerns. Engineers from John Snow Labs

found⁠

that a predictive readmission model that was trained, optimized and deployed at a hospital would start sharply degrading—and predicting poorly—within two to three months. A similar case occurred with IBM Watson’s oncology treatment recommendation service, where it was

withdrawn from the market⁠

after it began recommending unsafe treatments after experiencing model degradation.

This toolkit is meant to provide a set of educational resources for understanding healthcare AI products, provide a tool that helps generate procurement RFP contract language that hold AI companies accountable for their product outcomes, and outlines ways in which procurement organizations can monitor and evaluate the performance of healthcare AI products over time.

The resources in this toolkit are iterative and constantly evolving — we welcome contributions and feedback to this work. Feel free to reach out to us at healthytechbias@gmail.com.

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.

Health Tech Equity

Healthcare Procurement Guide:

How to write healthcare AI RFPs and contracts designed for privacy, bias prevention, intellectual property, technical transparency.

Table of Contents

How to use this Toolkit:

Introduction