Skeptically Optimistic: An AI Ethicist on Using AI for Your Health
Ask Jess Morley whether she is worried about AI and she will not give you a clean answer, because she does not think there is one. At the macro level, yes — she thinks we are systematically overestimating what these systems can do and underestimating the harm they cause. Drill down to specific use cases and her unease gives way to something closer to enthusiasm. The position she lands on is "skeptically optimistic": good about the opportunities, conscious that we are on the wrong path.
Part of The Agentic Patient, a Faces of Digital Health series on how patients actually use AI — which tools, which prompts, which safeguards. This episode is a conversation between host Tjaša Zajc and Dr Jess Morley, Associate Research Scientist at the Yale Digital Ethics Center and a former AI subject-matter expert at the UK Department of Health and Social Care.
What does the wrong path look like? In Morley's framing, AI in healthcare has become a hammer in search of nails. Governments lean on health systems to adopt it; the companies building the models lean from the other side. But very little of that pressure is needs-led. What gets built tends to be whatever was easiest to build — which usually means wherever the data already sat — rather than an answer to a problem clinicians or patients were actually stuck on. The alternative she would prefer is unglamorous and well understood: start from a difficult, specific problem and ask whether AI can help, instead of starting from the tool and looking for somewhere to point it.
That gap between what is legally permissible and what is socially acceptable runs through the whole conversation. Morley uses the DeepMind–Royal Free episode as the cautionary case. UK law sorts data use into broad purposes — direct care, service analytics, research — each carrying different consent requirements, and direct care does not require explicit patient consent to share a record. The logic is mundane: in a hospital, you should not have to re-consent every time a second clinician picks up the clipboard. In the data and AI space, she argues, those categories collapse into one another, which is precisely why law alone is not enough. Law moves slowly and leaves gaps; hard rules can make things riskier rather than safer by giving people seams to slip through. Ethics is what lets you ask should we rather than could we — and, she notes, it is also what gets gamed, through "ethics shopping," where people pick whichever interpretation of a principle is easiest to comply with.
The principles themselves are worth revisiting, in Morley's view, but for a reason that cuts against how they are usually invoked. The familiar bioethics quartet — beneficence, non-maleficence, autonomy, justice, sometimes with explainability bolted on for AI — was designed to start a conversation about one clinician and one patient. AI's harms do not work that way. A diagnostic tool or a chatbot is trained on hundreds of other people's data and is doing the same thing to hundreds of people at once. The risks that surface there are bias, discrimination, group-level harm — and Western ethical traditions, built around the individual, are short of frameworks for thinking about population-level damage that concentrates in the most vulnerable.
Her sharpest example is the ambient scribe, which she insists has been mis-sold. It is pitched as a very good transcription service, but transcription has existed for years; this is an inference service, making decisions about what gets recorded, what is weighted, and how it is coded into structured data such as SNOMED or ICD-10. Those coding decisions can be gamed, and they have consequences. Morley points to records inflated for billing, to a diagnosis miscoded in ways that could expose a patient to legal or criminal risk, and to a quieter failure mode: a scribe that logs a test as ordered because ordering it was the logical next step in the consultation — so the test that was never actually requested never gets done. None of this, she notes, was meaningfully trialled before deployment.
If the tools are flawed, access to them is unequal in a way that compounds the problem. Morley expects a bifurcation: people who can reach a human, people who can pay for the best-performing model, and people left with the weakest free versions — or with nothing. And because models increasingly learn from their own outputs, the old "garbage in, garbage out" worry returns in a more serious form.
So why do patients turn to AI at all? Not, Morley and Zajc agree, because they would rather talk to a chatbot than a doctor. They use it because the doctor is 18 months down a waiting list. Which is what makes the practical question — how do you use these tools well — the heart of the episode.
How to use AI for your health: Jess Morley's guardrails
Use them, but know what for. Talking through a list of symptoms, generating questions to ask your clinician, or understanding what a term means is reasonable — not so different from "Dr Google." The difference, and the risk, is that generative AI is far more persuasive than a search engine.
Don't take anything at face value. These systems are stochastic, so they give different answers on different runs; they are trained to be sycophantic, catering to you in a way a doctor will not; and sounding confident is not the same as being accurate.
Behave like a toddler who can only say "why." Interrogate every recommendation. You've told me this — why? Never drop the why.
Use it to aid your thinking, not to replace it. The danger is overreliance with no questioning of what it returns.
Be specific. Vague questions get vague answers. Give time, location, context, and how things have changed. Asking the model to generate 20 follow-up questions before it assesses you tends to produce a better result.
Don't lean on it for diagnosis. Morley rates these tools poorly at diagnostics. They are more useful for "what should I ask my clinician" or "what might I be missing" than for telling you what you have.
Know when to walk away. Like social media, these systems are engineered to keep you engaged — notice how often a reply ends by asking you another question. For anyone prone to health anxiety or OCD, that engagement loop is itself a risk.
Watch the context window. The longer a single chat runs, the more performance degrades. Start a fresh chat, switch models, or copy your history into a new document when a conversation gets long.
Get the model to write its own prompt. Ask it to write you a prompt for what you want to know, then run that prompt in a fresh chat — or a different model entirely. It is one way around a literacy gap.
Red-team your own assumptions. Ask the model to argue against you, and play models off each other (handing one model's answer to another to fact-check). Skip this one if you are prone to medical anxiety.
Ask for success criteria. Tell it to define what a good answer looks like and only return one once it meets those criteria, so it keeps iterating.
Ask for a confidence score. A simple "how confident are you in that, one to ten?" is a quick reliability check.
Build a "harness." Use skills and apps so the model works from your own history, preferences and requirements rather than the internet's average. A prompt as plain as "tell me how I can harness you to work better for me" will get you started — and the model will explain how.
The argument underneath the tips
Morley's pragmatism sits on top of a firmer position: tools that give medical advice — which general chatbots categorically do, whatever their makers intended — should be regulated as medical devices and held to the standards that implies. Absent that, she says, treat it as a buyer-beware market and remember you are in unregulated space. On deployment, she sees real promise in building bespoke models on top of a hospital's own protocols and guidelines, constrained from straying outside them — but that needs in-house skill that is largely missing. And she pushes back on the assumption that feeding a model every medical textbook ever written will let it reason like a clinician. Much of medical expertise is reasoning to the best explanation from experience; much of it is never codified at all, which is exactly why the "last mile" stays unsolved.
The episode closes on a worry Zajc raises from a therapist she spoke to: that AI, by agreeing with us, narrows our viewpoint, leaving the clinician to work backwards from what the patient already discussed with a chatbot. Morley's response doubles as the throughline of the whole conversation. A clinician asking a model "here are the symptoms, I think it's X, what am I missing?" is using it well — that is differential diagnosis, and it is healthy. Turning off your own brain and treating it as a searchable textbook is not. The risk worth guarding against, for patients and doctors alike, is the same: outsourcing the thinking.