An AI a Day Keeps the Doctor Away?: Regulating Artificial Intelligence in Healthcare
Artificial Intelligence (“AI”) promises to transform many aspects of everyday life for Canadians. AI tools are predicted to dramatically improve the provision of heath care by improving the quality, safety, and efficiency of diagnostic tools, treatment decisions, and care. Although AI innovations are, in many cases, still years away from general deployment into the Canadian health care ecosystem, AI is already used in some circumstances to read medical images, allowing machine learning to support diagnosticians in their decision-making.
Like many other jurisdictions, Canada’s health governance systems currently lack the appropriate legal and regulatory mechanisms to effectively deal with the challenges that AI poses. There is currently uncertainty with respect to key issues such as the related legal requirements for health privacy, medical device regulation and liability for AI-related harms. In Canada, regulation of AI in health care involves the additional challenge of navigating constitutionally fragmented jurisdiction over health care, which results in layers of governance and the need to coordinate multiple different actors.
This blog post highlights some of the legal challenges and issues that need to be addressed in order for Canada to have a robust and well-regulated governance structure for the use of AI in health care, including:
- Coordination of federal and provincial authority;
- Privacy and oversight with respect to the use of AI in treatment;
- Promotion of Equity through AI; and
- Liability for AI-related harms.
Coordination of Federal and Provincial Authority
Canada’s federal system and constitutional division of powers pose unique challenges for the regulation of AI in health care. Under the Constitution, health care is under provincial jurisdiction. Although similar, each province has its own set of regulatory frameworks addressing the safety and quality of health care, health information privacy, informed consent, human rights and non-discrimination, and licensing of health care professionals. With respect to the adoption of AI, provincial legislation and regulation will be the primary legal structure that governs the end users of AI technology and its application to patients.
However, despite health care being primarily a provincial concern, the federal government plays a significant role, particularly through its spending powers under the Canada Health Act, and its responsibilities for Indigenous Peoples, federal prisoners, and the military. The federal government is also a significant player in the regulation of drugs and medical devices.
Health Canada is the key regulatory authority at the federal level that controls which medical devices are available for sale and may be included in the public insurance plans of the provinces and territories. Health Canada’s primary mode of regulation is through the licensing process applicable to all medical devices. This licensing process requires manufacturers to classify their devices according to risk (e.g., invasiveness, risk of erroneous diagnosis, and intended medical purpose) under the Medical Devices Regulations, and obtain approval from Health Canada. If a device is licensed, the Medical Device Directorate continues to monitor the safety and efficacy of the device.
A significant challenge for the licensing and regulation of AI-technology is the application of machine learning, which is often referred to as “black-box” decision making, because the relevant algorithms are often proprietary and commercially sensitive and decisions and impacts of the algorithms cannot be fully explained. A question that is being asked by regulators around the world is “how can a regulator verify and validate machine learning algorithms to ensure that they do what they say well and safely?” Another question is: what role should machine learning and automated decision making have in health care?
Other key actors in the regulatory framework of Canadian health care are the professional bodies that provide oversight and self-regulation. Ensuring coordination between these regulatory bodies, the provincial and federal legislatures, and Health Canada to minimize or eliminate regulatory blind-spots will be a challenge that must be overcome to ensure good governance of AI in health care.
In April 2021, the European Commission released a 108-page proposal to regulate AI. Although the European Union has yet to reach consensus on the final text of the legislation, the proposal has received significant interest, and the question of how the European model could inform the development of AI governance in Canada is being considered by thought leaders in Canada. 
Privacy and Oversight
Even though the European Union has not yet implemented an overarching framework for AI regulation, the rules in its General Data Protection Regulation (“GDPR”) provide significant guidance for the European medical community with respect to regulation of medical AI. For example, under the GDPR, any controller of an AI-system based solely on automated processing must provide the subject with information about the existence of the automated decision-making, meaningful information about the logic involved, and the significance and consequences of such processing. The GDPR also provides a robust regulatory framework that governs the data privacy of citizens whose data may be used in machine learning algorithms. In Europe, the GDPR also requires that medical AI have human oversight. Further, the training data for the AI must be checked for bias and the ongoing operation of AI must be constantly monitored for the occurrence of bias to ensure that use of AI does not unintentionally result in discrimination.
Recent changes to Canada’s provincial privacy landscape suggests that Canada will not only follow the European example, but also seek to enforce robust privacy rights in its own way. For instance, on September 22, 2021, the province of Quebec’s landmark legislation, the Act to Modernize Legislative Provisions respecting the Protection of Personal Information ("Bill 64"), received Royal Assent. Bill 64 will impose a duty to inform with respect to technological tools that enable the identification, location or profiling of an individual in order to collect personal information from an individual. From September 22, 2023, organizations will also be required to inform the individual when a decision is made based solely on automated processing of his or her personal information, no later than the time the organization informs the individual of that decision. Organizations shall also give the individual the opportunity to make representations to a member of their staff who is in a position to review the decision. For more information about Bill 64, consult our blog series here.
Similarly, the federal government’s Directive on Automated Decision Making (“Canada ADM Directive”), indicates that Canadian regulatory frameworks will also likely require that any health-focused AI technology provide for human intervention in the decision-making process, and ensure that all data be tested for bias and non-discrimination. The Canada ADM Directive is a risk-based governance model that establishes four levels of risk, judged by the impact of the automated decision. Certain risk-mitigating requirements are then established for each impact level, including: notice before automated decision making decisions and explanations after automated decision making decisions; peer review; employee training; and human intervention.
Promotion of Equity
Arguably the most significant concern associated with the use of AI and automated decision making in health care is their potential to amplify bias and discrimination. Canada’s health care system already grapples with the problems associated with inequities in health care – from differential resource allocation between communities, to differential treatment of individuals based upon their gender or race.
The current legal framework for funding health care in Canada (i.e., the Canada Health Act) only protects universal coverage for “medically necessary” hospital and physician services. Due to the current novelty of AI-assisted medical services, it is unlikely that many AI-assisted medical services would currently be considered medically necessary. Therefore, only patients who have the means to afford add-on fees or private or boutique health care would gain access to this sophisticated technology. Further, if only larger medical centers have the infrastructure and access to such computer science programs necessary to develop AI-assisted medical programs, access to AI-technology may be limited, even if cost is not a barrier. Therefore, regulating AI through the appropriate legal frameworks to ensure that it is developed and deployed in an accessible manner will be an important matter for legislatures to consider and address.
In addition to potential inequities of access, there are two main sources of concern relating to discrimination in AI systems: (1) bias in the data used to train the system; and (2) bias in the algorithm.
If the data used to train the AI system is flawed or incomplete, for example by failing to include sufficient data from a certain population, the AI system may be ineffective or dangerous for patients of the underrepresented population. For example, AI-assisted cancer screening tools that are trained primarily on images of light-skinned patients are more likely to misdiagnose cancer lesions in patients with skin of colour. Bias in the AI-training data, while easier to identify, poses serious questions relating to access to data, data transfer, and consent. The importance of training AI systems with data from diverse populations will have to be balanced with laws relating to data collection, use, transfer and storage across multiple jurisdictions, so that diverse patients residing in areas with less diverse populations are not at risk of being harmed by treatments based on a lack of diverse data.
Bias in the algorithm of an AI system may be impossible detect, particularly where machine learning techniques are employed. When the decision making of the AI is a “black box”, due to the opacity of how the AI is identifying patterns (and potentially to changes in the algorithm over time as the machine learns), it can be a challenge to ensure that discrimination is not occurring. To combat this, several jurisdictions are considering explicit legislative commitments that ensure AI systems are compliant with anti-discrimination and human rights legislation. Paired with robust monitoring requirements, these types of provisions would provide greater legal certainty, accountability, and public confidence in AI-assisted health care.
Liability for AI-Related Harms
Another considerable hurdle in the adoption of AI technologies in health care, particularly in the medical community, is the continued uncertainty regarding the potential liability attached to the use of AI. Who do you sue when AI goes wrong?
Although AI has been used for various applications over the past few years, it remains unclear where liability should fall when an AI system fails. In 2020, the question of who (if anyone) is liable when an AI-powered trading investment system causes substantial losses for an investor, was before the English courts for the first time. Unfortunately for the development of tort law in this important area, the parties reached an out of court settlement, leaving the question to be answered another day.
In Canada, medical harms may be dealt with under the law of negligence. How AI-technology changes the standard of care expected of a medical practitioner and, in particular, the acceptable level of decision-making delegation to the AI system, are questions that must be considered. Further, the courts must query whether an AI company has any liability to a patient that is misdiagnosed. Could an AI company contract out of its liability to a hospital, if harm results from the use of its technology? Is an AI company only liable if bias is found in the data or algorithm? What type of consent is needed from patients before AI technologies are employed?
Another area where AI may cause significant harm to Canadians is through breaches of private health care data, which could harm the public’s overall confidence in the health care system. In the recent Supreme Court of Canada decision, Reference re Genetic Non-Discrimination Act (2020 SCC 17), the majority held that the federal government had the power to make rules combating genetic discrimination and protecting health through its jurisdiction over criminal law:
Many of this Court’s decisions illustrate how the criminal law purpose test operates. A law directed at protecting a public interest like public safety, health or morality will usually be a response to something that Parliament sees as posing a threat to that public interest. For example, prohibitions aimed at combatting tobacco consumption and protecting the public from adulterated foods and drugs were upheld because they protect public health from threats to it...
Parliament took action in response to its concern that individuals’ vulnerability to genetic discrimination posed a threat of harm to several public interests traditionally protected by the criminal law. Parliament enacted legislation that, in pith and substance, protects individuals’ control over their detailed personal information disclosed by genetic tests in the areas of contracting and the provision of goods and services in order to address Canadian’s fears that their genetic test results will be used against them and to prevent discrimination based on that information. It did so to safeguard autonomy, privacy and equality, along with public health. The challenged provisions fall within Parliament’s criminal law power because they consist of prohibitions accompanied by penalties, backed by a criminal law purpose.
This case demonstrates that the federal government’s regulatory power is not limited to health care spending. However, it is unclear how effectively criminal law can be used to govern AI and, as AI technology becomes more common in health care, the legislatures and the courts will have to carefully consider how the current private and criminal law frameworks can be adapted to deal with attributing and apportioning liability arising from AI decision-making.
AI poses both risks and opportunities in the health care space. AI systems aim to democratize health and provide superior patient care. However, regulators must contend with the challenge of ensuring that AI technology does what it is intended to do, does it well, and that there remains legal accountability for any harms caused.
 Colleen M. Flood and Catherine Régis, Régis, Catherine and Flood, Colleen M., AI and Health Law (February 1, 2021). in Florian Martin-Bariteau & Teresa Scassa, eds., Artificial Intelligence and the Law in Canada (Toronto: LexisNexis Canada, 2021), Available at SSRN: https://ssrn.com/abstract=3733964.
 Canada Health Act, R.S.C., 1985, c. C-6.
 Medial Devices Regulations, SOR 98/282
 W. Nicholson Price II, “Artificial Intelligence in Health Care: Applications and Legal Issues” (2017) The SciTech Lawyer 14:1; David Schneeberger et al., (2020) The European Legal Framework for Medical AI. In: Holzinger A., Kieseberg P., Tjoa A., Weippl E. (eds) Machine Learning and Knowledge Extraction. CD-MAKE 2020. Lecture Notes in Computer Science, vol 12279. Springer, Cham. https://doi.org/10.1007/978-3-030-57321-8_12
 Law Commission of Ontario, “Comparing European and Canadian AI Regulation” (November 2021). https://www.lco-cdo.org/wp-content/uploads/2021/12/Comparing-European-and-Canadian-AI-Regulation-Final-November-2021.pdf
 Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) [GDPR]
 GDPR, ibid, Arts 13, 14.
 GDPR, ibid, Art 22. Scheenberge, supra note 5, 211.
 Schneeberger, supra note 5, at 211.
 Canada ADM Directive.
 Canada Health Act, R.S.C., 1985, c. C-6., s. 2 sub nom “hospital services” and “physician services”
 Adamson AS, Smith A. Machine Learning and Health Care Disparities in Dermatology. JAMA Dermatol. 2018 Nov 1;154(11):1247-1248. doi: 10.1001/jamadermatol.2018.2348. PMID: 30073260.
 Law Commission of Ontario, “Regulating AI: Critical Issues and Choices” (April 2021) at 38-39. https://www.lco-cdo.org/wp-content/uploads/2021/04/LCO-Regulating-AI-Critical-Issues-and-Choices-Toronto-April-2021-1.pdf
 Minesh Tanna “AI-powered investments: Who (if anyone) is liable when it goes wrong? Tyndaris v VWM” (November 2019). https://www.simmons-simmons.com/en/publications/ck2xifd2ddmrq0b48u46j2nns/ai-powered-investments-who-if-anyone-is-liable-when-it-goes-wrong-tyndaris-v-vwm
 Jeremy Kahn, “Why do so few business see financial gains from using A.I?” (October 20, 2020). https://fortune.com/2020/10/20/why-do-so-few-businesses-see-financial-gains-from-using-a-i/
 In Quebec, civil law liability principles would govern.
 Mélanie B. Forcier, et al., “Liability issues for the use of artificial intelligence in health care in Canada: AI and medical decision-making” (July 2020) Dalhousie Medical Journal 46(2). DOI:10.15273/dmj.Vol46No2.10140
 Reference re Genetic Non‑Discrimination Act, 2020 SCC 17 at paras. 73, 103