reveals that government agencies purchasing algorithmic systems often have no input into — or even knowledge about — their design, or whether how they operate aligns with public goals and values. Purchasers often have no access to the internal rules that algorithmic systems use in order to make predictions, or what datasets were involved in the machine learning process. Algorithmic systems recommend and automate many critical pieces of government decision-making, effectively promoting computer code to policy.
Moreover, designing algorithmic systems involves making decisions about goals, risk tolerance, and values that ultimately impact the outputs of the predictive models. Especially when these systems are adaptively and continuously learning, the design choices made by software developers ultimately carry out policy unilaterally as the algorithmic system changes over time. Yet procurement systems
for enforcing transparency and accountability when purchasing algorithm-based systems.
The requests for proposal (RFP) templates provided here offer a mechanism for soliciting transparency and accountability from algorithmic system providers by defining clear terms for responsible AI development. The goal is to help agencies generate language that they can insert into their own RFPs.
How to Use the Tool
This tool generates templates for procurement language that incorporate the World Health Organization guiding principles for healthcare AI, which will be further outlined in the Healthcare AI 101 section. These principles include: technical transparency, inclusiveness and equity, ensuring product quality, privacy protections, sustainable products, and promoting accountability for algorithmic impacts. The tool provides document rider clauses for procurement officers that can be plugged into requests for proposals. These clauses can provide the starting point for further customization according to your agency’s specific acquisition needs.
The process is simple.
Enter your organization’s name. This will fill in the document with your institution’s title.
Select the disclosure timeline setting for the vendor to notify your organization about changes to the algorithm.
Provide a contact email address and a website for hosting this policy. This will insert contact details into the document.
That’s it! The document will be generated and available for copy-paste or download via HTML or Markdown file.
Limitations of the Tool
The language produced by this tool will likely require additional customization to suit your specific use case. For instance, you should be prepared to answer questions such as:
Who will own oversight of the procurement contract?
What metrics will be used to measure the performance of this AI tool? Do these metrics connect to desired outcomes within the impacted patient population?
What is an acceptable performance threshold in the monitoring metrics for considering whether to retire the tool or require changes?
Rationales for RFP Template Clauses Within This Tool
: Having a reasonable explanation for why a recommendation is being made is a crucial component of being able to trust an AI tool. Within medical care delivery, there should be a clear, logical rationale for taking actions that can impact patient health outcomes. AI tools develop their own internal rules, based on data fed into them during their training phase. These rules should be exposed to auditors, tool users, and patients in order to give human operators the confidence to accept or reject recommendations. Thus, the template recommends that algorithm-based systems be required to provide comprehensible explanations to human operators about why a particular decision was made, along with a list of the most influential variables that led to the decision.
: Healthcare work is a high-impact domain that has the potential to inflict severe privacy, health, and financial burdens on patients. Consequently, AI-based healthcare tools must be designed to avoid and mitigate the impact of social biases on recommendations used in medical care delivery. There is also a legal responsibility to avoid violating U.S. anti-discrimination laws by limiting prejudiced behavior based on protected attributes like race, gender, religion, etc. Fairness and equity are subjective concepts within algorithmic systems, and often depend on context to be correctly defined.
To solve this, AI tools should have the ability to accommodate the complexity of algorithmic fairness by continuously monitoring multiple metrics that capture different aspects of equity. The template recommends establishing a process for defining multiple fairness metrics that are customized for the use case of the AI tool, as well as a monitoring system with defined alert thresholds for flagging potentially biased behavior.
: Medical devices and software need to have high levels of accuracy, reliability, and availability in order to avoid negative impacts on patient health outcomes. This extends into AI tools that are used within a patient’s medical decision-making process. However, these tools can be complex to govern over time. Algorithms have the ability to evolve as updates and changes are iteratively applied by developers when, for example, adding new functionalities, improving or maintaining model performance, or patching security vulnerabilities.
To address the need for standards and the evolving nature of algorithmic products, the template recommends establishing a set of service-level agreements (SLAs) that define contractual business requirements for the technology, as well as algorithm change protocols (ACPs) that track developers’ anticipated future changes to the algorithm.
: The HIPAA “safe harbor” clause that allows organizations to share and sell de-identified health data has been shown to be vulnerable to potential re-identification of patient data. The safe harbor clause applies when health data has been stripped of 18 defined types of patient-identifying variables, but third-party datasets and artificial intelligence techniques
on the sale of de-identified personal health information, as defined in the California Consumer Privacy Act (CCPA). If this restriction is infeasible for the contract, the template recommends the deletion of identifiable fields as opposed to anonymization, to reduce the risk of re-identification of patient data.
: A combination of factors — including the opacity of algorithm logic, lack of access to the technical source code and datasets, and a high barrier of technical literacy — prevents patients or physicians from being effective advocates for the impacts of algorithms on patient communities.
The template recommends contractually requiring third-party audits by experts who have transparent access to technical resources within the contractor’s systems. The third-party auditors should be paid by the procurement organization in order to avoid conflicts of interest between the auditor and the contractor. To be sensitive to contractor concerns around intellectual property, non-disclosure agreements can be enforced with the third-party auditor to prevent the leakage of trade secrets.
: A framework for continuous monitoring and evaluation should be in place in order to guarantee the performance quality of the algorithmic system.
In order to identify success and failure states for the tool, the template recommends establishing a set of monitoring metrics and acceptable threshold bounds that are agreed to by the procuring organization and the algorithm contractor. There should be clear warnings when this threshold is being approached, and a defined remediation policy should be ready for activation. A communication plan should also be in place to ensure the contracting agency is informed in advance of any updates or changes to the tool.
How to Contribute
To contribute templates and sample procurement language, feel free to email firstname.lastname@example.org.
For code contributions, this tool’s source code is open-source, and we welcome pull requests to add additional functionality to the app!
This project was inspired by the City of Amsterdam’s set of sample procurement clauses for AI governance, as well as by the disclose.io project, which inspired the structure of the open-source contract templates.
Now that you have generated your RFP template, head over to the