Skip to content
HealthyTechBias
Health Tech Equity
Healthcare AI 101
Procurement Language Generator
Fairness Metric Quiz
Responsible AI Product Design
Glossary
More
Share
Explore

icon picker
Healthcare AI 101

Everything you need to get up to speed with healthcare AI tech.
⁠

Healthcare AI Pillars

Explainable & Transparent Products
Physicians and healthcare algorithm operators should be able to provide transparent and comprehensible explanations for patients regarding algorithms involved in care delivery.
Fair & Unbiased Healthcare Delivery
Researchers and journalists have called out an increasing number of racial bias factors within healthcare algorithms, including: racial gaps in
patient care utilization
⁠
and costs,
race-based correction factors
⁠
for kidney disease diagnosis, and new sources of patient data collected by data brokers, uncovered by HIPAA, enable social determinants of health and lifestyle factors to
impact insurance premiums
⁠
.
Robust Performance across Different Groups
Different socio-economic environments and disease groups respond differently to medical treatment — interventions should be tailored to the group as specifically as possible.
Tight Privacy & Security
⁠
HIPAA requirements
⁠
enforce strict guidelines on personal health information that must be anonymized as well as cybersecurity protocols for healthcare data.
Names (Full or last name and initial)
All geographical identifiers smaller than a state, except for the initial three digits of a zip code if, according to the current publicly available data from the U.S. Bureau of the Census: the geographic unit formed by combining all zip codes with the same three initial digits contains more than 20,000 people; and the initial three digits of a zip code for all such geographic units containing 20,000 or fewer people is changed to 000
Dates (other than year) directly related to an individual
Phone Numbers
Fax numbers
Email addresses
Social Security numbers
Medical record numbers
Health insurance beneficiary numbers
Account numbers
Certificate/license numbers
Vehicle identifiers (including serial numbers and license plate numbers)
Device identifiers and serial numbers;
Web Uniform Resource Locators (URLs)
Internet Protocol (IP) address numbers
Biometric identifiers, including finger, retinal and voice prints
Full face photographic images and any comparable images
Any other unique identifying number, characteristic, or code except the unique code assigned by the investigator to code the data
Data-sharing and Interoperability
FHIR standards
Data trusts and other kinds of data-sharing collectives

⁠
image.png
⁠
⁠
Healthcare use cases: patient engagement, care delivery, population health, R&D, and administration
The benefits of machine learning in healthcare include:
Achieving early intervention with at-risk patients, improving the effectiveness of treatment.
Providing insights into health trends in large datasets to enhance public health policies and engagement.
Automating administrative processes to save on resources and improve the workload of healthcare workers.
As a diagnosis tool, often through image categorization techniques.
Enhancing and accelerating the research of new drugs, treatments, and diseases.

What are unique challenges for health technology?

Highly complex biological/environmental systems in medicine that make causal inference difficult.
Meaningful patient consent can be difficult given the complexity of healthcare + algorithms
Fragmented EHR adoption has made it still difficult for IT to deliver easy transparency to patients
High risk outcomes in patient care make healthcare providers more risk-averse in adopting technology solutions.
Privacy + security are regulated by hefty HIPAA fines and impose more regulation in healthcare than in other industries.
High diversity between patient populations in different geographical and socio-economic environments have very different medical needs and responses, challenging the generalizability of AI models.
Healthcare studies and research typically take longer periods of time to complete cycles to comply with patient safety and medical efficacy validation requirements.

How is health data different from other data domains?

Machine learning in a clinical setting presents unique challenges that complicate the use of common machine learning methodologies. For example, diseases in EHRs are poorly labeled, conditions can encompass multiple underlying endotypes, and healthy individuals are underrepresented. Most data, such as clinical notes, are created by human judgment, and encode many of the human creator’s underlying biases.
It is important to make a distinction between the various sub-areas of healthcare, such as research and development (products such as eHealth apps, wearables, MRI scanners, smart medicine), professional care providers (primary care, drug distribution, complex surgery) and recipients of care (patients). Because each sub-area has different needs.

Privacy

Healthcare data should be anonymized before being used for secondary purposes that go beyond the delivery and administration of patient healthcare. Ensuring patient privacy is becoming increasingly difficult given the volume of new personal information being shared and stored online with increasing digital healthcare adoption.
Even after records are stripped of personally identifying information, sophisticated algorithm techniques have the ability to infer PHI from anonymized data and represent a persistent privacy risk. Anonymized data can also be less useful for analysis when patients are more difficult to track across datasets and institutions. Anonymization is a legal requirement and an operational necessity in healthcare data products, but brings its own sets of tradeoffs in the performance and transparency of algorithmic products.
However, a recent
study
⁠
led by researchers at the University of California warns that current anonymization techniques fail to guarantee that individual patients cannot be re-identified. This re-identification is possible through the usage of other datasets that share a common identifier or through AI extraction techniques.

Consent

Obtaining informed consent in the digital age is an important tool in protecting patients’ rights, but has become increasingly difficult given the complexity of the medical supply chain. Providing this education to patients requires precious clinical time from physicians to explain the underlying systems involved in a medical diagnosis or decision. Physicians have been reluctant to disclose the involvement of algorithms in medical decision-making to patients, seeing it as an obstacle to their job that raises more trouble than it’s worth.
When considering the expansion in health data sharing, the physician may also not be aware of the extended network of institutions that have access to a patient’s data and how healthcare algorithms may have utilized that information (either for building the algorithm or in producing a prediction). The way that advanced analytics products have evolved often uses data to iteratively explore many potential use cases, consequently making it difficult to reliably inform patients on how their data is used. Even when these use cases are identified, the underlying technical systems are so complex that it puts a large burden of scientific/technical literacy on the patient to understand the benefits and risks. This literacy gap and informed consent problem is even more exacerbated for vulnerable populations.

Trust

Patient trust is a tricky thing to define, being a combination of transparency and accountability, but has critical gaps within current healthcare institutions. Strong legislative guidelines like HIPAA and regulatory bodies like the FDA ensure high quality, secure healthcare and medical devices. Hospitals and physicians are generally trusted stewards for protecting patient interests, but health data collected by tech industry companies currently falls into a regulatory gray zone. This class of sensitive yet unregulated health data consists of behavioral information like social media post activity, internet search history, and wearables vitals data that are able to infer medical information about patients but are not subject to health privacy laws.
Research Ethics Boards (REBs) at tech companies like Facebook’s Internal Review Board often face challenging internal politics when holding their own institutions accountable. Tech workers commonly do not receive education programs on handling sensitive health data and healthcare providers are not always given the resources they need to protect patient privacy and rights. To promote meaningful trust for patients, the healthcare industry needs to find answers to these issues.

Data Governance


The process of building a technology platform involves significant work including data collection, storage, analysis, monitoring, and sharing. These pipelines can involve many different parties in both production and consumption, including patients, developers, physicians, advocacy groups, academic researchers, and regulatory government bodies. No one institution can manage and oversee the whole process, which leads to accountability in this space often falling through the cracks. For information asymmetry and process transparency reasons, patients/providers are unable to manage the entire healthcare/technology pipeline. The problem of informed consent is present in many other technology industry domains, but health tech pipeline transparency is often extra pronounced due to the degree of siloing between the different institutions.

Liability


A key consideration for liability in AI systems is that the number of people who are involved in medical decision-making for a patients care is multiplying. The black box nature of certain AI models and the complexity/technical literacy involved in AI explanations also makes it difficult to explain exactly how an AI system comes to a decision or prescribes a treatment for determining liability. Is the issue with the encoded rules and biases in the algorithm or with the medical provider that interprets those recommendations in the context of their medical training?
This isn’t an entirely novel situation. Dr. Danton Char, a Stanford anesthesiologist, compares AI to a prescription drug. Char says, “While clinicians can’t be expected to understand every biochemical detail of the drugs they prescribe, they at least need to know that they’re safe and effective, based on their clinical experience and knowledge of the medical literature. As for AI systems, I wouldn’t use one unless I’m confident after careful research that it’s the best option. You don’t want to put anyone’s life at risk because you haven’t vetted the tool you’re using.”
⁠
Humaaans - Wireframe.png
⁠
⁠

What does the regulatory landscape look like for health data?


American Medical Association
In June 2018, the American Medical Association (AMA) issued its first guidelines for how to develop, use and regulate AI. (Notably, the association refers to AI as “augmented intelligence,” reflecting its belief that AI will enhance, not replace, the work of physicians.) Among its recommendations, the AMA says, AI tools should be designed to identify and address bias and avoid creating or exacerbating disparities in the treatment of vulnerable populations. Tools, it adds, should be transparent and protect patient privacy.
⁠
Screen Shot 2021-10-19 at 5.35.03 PM.png
⁠
⁠
The FDA’s risk categorization framework for algorithms in medical devices. In a medical context, risk is calculated according to the degree to which algorithms control medical decision-making.
U.S. Food & Drug Administration
Perhaps the most radical ML property from a regulatory perspective is the potential for algorithms to evolve after release and distribution. This capability is what is referred to as continuously learning systems. Currently, this is only a theoretical concern as there is a blanket prohibition of this scenario across every existing and proposed TPLC regulatory framework. Might there come a time when this prohibition will be perceived as imposing an undue constraint on innovation? Is there a scenario – perhaps in a robotics context – where allowing an initial set of SaMD instances to evolve wholly independently from one another will be identified as an absolute requirement? How would today’s notions of manufacturing lifecycle and quality need to adapt?
The FDA's proposed Total Product Lifecycle (TPLC) regulatory framework is designed to ensure ongoing ML algorithm changes are:
Implemented according to pre-specified performance objectives,
Follow defined algorithm change protocols
Utilize a validation process that is committed to improving the performance, safety, and effectiveness of AI/ML software
Include real-world monitoring of performance.

⁠
Screen Shot 2021-10-19 at 5.48.52 PM.png
⁠
⁠

Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.