AI Ethics and Governance in Adult Social Care
AI is being deployed across health and social care at pace, but the UK does not currently have a single, AI-specific regulatory framework, and the existing frameworks were not designed specifically with these tools in mind. In this session from the Connected Health & Care Summit 2026, Stewart Duffy, health technology and regulatory advisor at Weightmans, maps the full legal and governance landscape that organisations must navigate before, during and after deploying AI, covering data protection, medical device regulation, CQC requirements, cybersecurity, professional accountability and the critical question of where liability sits when things go wrong.
Watch Stewart Duffy of Weightmans explain the full legal landscape for AI in health and social care, from automated decision-making rules through to medical device law, CQC requirements and the cybersecurity risks that LLMs and agentic AI introduce.
What the session covers and the key principles
This session provides a detailed overview of the regulatory and governance considerations for organisations deploying AI in health and social care. The key principles were:
- You cannot delegate accountability to the tools. Professional and organisational accountability remains with the humans who deploy and oversee them
- Solely automated decision-making using health data is prohibited under data protection law unless very narrow conditions are met
- "Meaningful human involvement” requires someone with the skills, knowledge and organisational authority to intervene in the decision and provide appropriate oversight of the process
- Software can be a medical device, and deploying an uncertified medical device is a criminal offence
- The UK has no AI-specific regulation, but existing frameworks covering professional conduct, CQC requirements, data protection and medical device law all apply directly
- Function creep is a foreseeable risk. AI scribes may be reviewed by clinicians on day one but not six months later
The regulatory landscape and what organisations must consider
Stewart Duffy maps out four layers of regulation. First, professional regulation, where codes of conduct require clinicians to exercise appropriate oversight when delegating tasks. He draws a direct analogy with GMC delegation guidance: if you would need to satisfy yourself that a colleague had the skills to carry out a task, the same due diligence applies when delegating certain tasks to a digital tool. Second, systems regulation through the CQC, where equipment must be secure, suitable for purpose and deployed in a way that mitigates foreseeable risk. Third, generic legislation including data protection, human rights and consumer rights. Fourth, NHS-specific requirements such as clinical safety standards, which are currently being consulted on for reform.
On cybersecurity, the session highlights that large language models remove the distinction between coding languages and plain English, meaning systems can now be jailbroken by anyone who can write a prompt. Agentic AI introduces further risk because it requires access rights across systems.
What this means in practice and who it is relevant to
The session is structured around practical defensibility. If an AI tool causes harm, the organisation will need to demonstrate that it went through an appropriate process to evaluate the technology, assess the risks and set controls before deployment. Stewart Duffy outlines the questions every organisation should be asking: is the tool making an automated decision, are we delegating a professional task, how will the output be used, and does our intended use match the manufacturer’s intended use?
On liability, Stewart Duffy notes that technology vendors typically use contractual terms to define and limit their exposure. The question of who bears accountability when AI displaces professional judgment is described as a hot topic, with the Medical Protection Society recently publishing a call to action on this issue. One possible model discussed in the session would insulate professionals from personal accountability provided the organisation met relevant standards in deploying the technology, with the system bearing the cost of negligence claims.
The session closes with practical takeaways: be deliberate and purpose-driven, develop an AI policy before deploying tools, ensure clarity of ownership and accountability, be realistic about what the tools can achieve, and do your due diligence.
This session is relevant to chief digital officers, medical directors, governance leads, legal teams, data protection officers, information governance leads, clinical safety officers, CQC registered managers and anyone responsible for technology procurement, deployment or risk assessment in health and social care.
Frequently asked questions
The session is presented by Stewart Duffy, a specialist health technology and regulatory advisor at Weightmans. He covers the full regulatory landscape for AI in health and social care, including data protection, medical device law, CQC requirements, professional accountability and cybersecurity. The presentation is followed by audience Q&A on topics including DTAC, future-proofing governance and liability.
Decisions solely resulting from automated processes involving health data are prohibited under data protection law unless one of three narrow conditions is met: explicit consent from the data subject, a contract with the data subject, or processing required or authorised by law that meets a substantial public interest condition. Stewart Duffy describes solely automated decision-making in healthcare as problematic and emphasises that “meaningful human involvement” requires someone with the skills, knowledge and organisational authority to intervene.
Yes. The session makes clear that software can be a medical device under existing regulations, and deploying an uncertified medical device is a criminal offence. Medical devices are classified by risk, from Class 1 (a walking stick) upwards. Under the EU AI Act, a medical device would also be classified as a high-risk system. The MHRA publishes guidelines on borderline products for cases where the classification is not straightforward.
The GMC guidance on delegation says clinicians must be confident that the person they delegate to has the necessary knowledge, skills and training. Stewart Duffy argues you can substitute “digital tool” for “colleague” and the same principle applies. However, the analogy breaks down because a tool does not take responsibility and cannot confirm that it understands the task. The GMC has not yet produced equivalent guidance specifically addressing delegation to digital tools.
Function creep is when a tool is used in a way that deviates from its intended purpose over time. Stewart Duffy gives the example of AI scribes. On day one, clinicians will review the generated notes before signing off. Six months later, they may stop checking because they have grown to trust the output. Organisations need to anticipate this risk and design controls that remain effective beyond the initial deployment period.
AI tools introduce a greater attack surface and novel forms of risk. Large language models remove the distinction between coding languages and plain English, meaning safeguards can be bypassed through prompt-based jailbreaking. Agentic AI requires access rights across systems to function, which introduces further exposure. The NCSC advises organisations to “approach adoption very carefully” and to ensure they understand what the tools are doing before deployment.
Stewart Duffy recommends a principles-based, technology-neutral approach, similar to the GDPR, which has shaped regulatory regimes globally since 2018 while providing a framework that can adapt as technology evolves. In health and social care specifically, the CQC regulatory regime and professional codes of conduct already provide risk-based frameworks that can apply to new technologies. He argues that healthcare is better positioned than many sectors because those frameworks already exist.
Be deliberate and purpose-driven in how you approach AI. Develop an AI and technology policy before deploying tools. Ensure clarity of ownership and accountability at both executive and clinical levels. Recognise that tools are one part of a complex system. Be realistic about what you want to achieve. And do your due diligence, both before and after deployment.
