F048 AI in healthcare Part 2/5: Radiology disrupted?

The general perception of radiology is that radiologists more or less look at medical images all day long. Eric Topol even joked radiologists could wear pajamas all day long, given that radiology departments are mostly located in basements.

So what exactly do radiologists do and how is AI impacting their work?

“Radiologists play an important role in every aspect of medical imaging: from advising doctors which exam to order, to performing exams, and in the end interpret them. Some radiologists also perform procedures, so the extent of work a radiologist would do depends on his specialization and profile. The training is long: in the United States, students go to a four-year medical school after college, followed by an internship in radiology,” explains radiologist Woojin Kim currently the Chief Medical Information Officer for Nuance. Woojin Kim is former Chief of Radiography Modality, Director of Center for Translational Imaging Informatics, Associate Director of Imaging Informatics, and Assistant Professor of Radiology at the Hospital of the University of Pennsylvania. He has also served in the past as the Interim Chief and Interim Fellowship Director of Division of Musculoskeletal Imaging and Chairman of the Efficiency Committee.

Enters AI

Woojin Kim.

“If I, who is single, would say I created a robot that can replace all mothers, you would think I am either an idiot, crazy, or both,” says Woojin Kim as a comment to the idea that AI will replace radiologists completely. As he further explains, radiologists don’t look at only one thing in an image, but hundreds if not thousands of things, which AI is incapable of at the moment.

Algorithms are very narrow; most AI models are based on retrospective studies, lacking real world validation, which is needed before wider adoption in clinical practice.

The problem of unstructured data

When thinking about AI applications in radiology, the focus is usually on image recognition solutions. However, says Woojin Kim, the real goldmine of data are radiology reports. The problem in report analysis, however, is a lack of structured data.

“Like in any other field, the standard 80/20 rule applies to data in radiology. 80% of the data is unstructured, mostly free text; only 20% of the data is structured. Perhaps some software solutions claim to structure data, but what they usually offer is templates with multiple drop-down menus that are not user friendly and in the end still produce a form of free text, which is from the database perspective unstructured,” says Woojin Kim.

Concept drift and AI brittleness

Two other issues challenging AI development and introduction in clinical practice are AI brittleness and concept drift. AI brittleness refers to the changing accuracy of AI models when they are transferred from one dataset to another. An AI model trained on a data set from one hospital can fail when transferred on a different data set in a different hospital, because of different machines used, different protocols doctors follow, etc. Additionally, what can occur even when a model works, is concept drift - a decrease in accuracy of an AI model over time due to imaging diagnostics changes (different device, new protocols, etc.).

In order to prevent that, AI companies would need to keep track of changes happening in hospitals. But since hospitals won’t automatically update companies on the changes they implement, keeping AI models updated is difficult. A partial solution to this problem would be user feedback based improvement. How to collect the feedback is an additional challenge, due to busy schedules doctors have. “It is unrealistic to expect direct feedback from doctors, at least not on a larger scale. Companies should gather information about their solution’s use with built-in analytics and analysis of the use of their apps. This requires partnerships with other IT solution providers to enable solution integration. To achieve any adoption, solutions need to become an integral part of doctor’s workflows, not an interrupting obstacle.

Radiology and dermatology were among the first fields targeted as obsolete in the era of AI. “A lot of the hype began in 2012 with convolutional large-scale visual recognition that drove an explosion of interest in advancement in computer vision. In 2016 Geoffrey Hinton who has been called The Godfather of the deep learning predicted radiologists would be replaced in 5 to 10 years in AI. Today the hype is gone,” says Woojin Kim.

Some questions addressed:

- The general perception is that radiologists sit in basements looking at pictures. What is the reality, what other assignments do they have?

- Because radiologists look at pictures, they were the first target of hyped assumptions that they will be replaced by AI, same as dermatologists and other doctors diagnosing on the basis of an image. Can you comment on that?

- AI models are brittle: accuracy might change in hospital B if the model is trained on data from hospital A. How is this issue being addressed?

- How can doctors use AI as decision support if the models can be inaccurate because they are used on a different dataset than the model was trained on?

- The consensus among those in the field is that AI will complement the work of radiologists because the combination of human + AI is the best. What are the best qualities each party brings to the final diagnosis?

- What is your opinion of the debates that radiologists will start to have more contacts with patients? There is a lively debate in radiology about whether or not that makes sense since the point of contact for radiologists is the patient’s primary care physician. Are there any potential problems coming from the fact that radiologists don’t know the full medical history of individuals?