A Literature Review on Deep Learning for Medical Image Diagnosis

Introduction

Medical imaging sits at the centre of modern diagnostic practice, and the volume of images now generated across radiology, pathology and ophthalmology has grown faster than the clinical workforce available to interpret them. Deep learning, a family of representation-learning methods built on multi-layered neural networks, has emerged over the past decade as the dominant computational approach to automating and augmenting this interpretation (LeCun, Bengio and Hinton, 2015). Where earlier computer-aided diagnosis systems depended on hand-engineered features designed by domain experts, deep networks learn hierarchical features directly from pixel data, a shift that has coincided with rapid and well-documented gains in diagnostic task performance (Litjens et al., 2017).

This review aims to synthesise the existing scholarly literature on the application of deep learning to medical image diagnosis, rather than to report any new experimental findings. Its scope is deliberately bounded. It concentrates on supervised deep learning applied to two-dimensional and volumetric diagnostic imaging in three well-studied clinical domains: radiology, pathology and ophthalmology. It considers how performance has been evaluated through benchmark studies, and it examines the recurring obstacles that the literature identifies, namely data availability, generalisation, interpretability and bias. Generative modelling, image reconstruction and treatment-planning applications fall outside the present scope. The review proceeds by first establishing the technical background, then thematically synthesising applications, performance evidence and challenges, before offering a critical appraisal of gaps and a short conclusion.

Background: Convolutional Neural Networks and Medical Imaging

The convolutional neural network (CNN) is the architectural backbone of most medical image diagnosis research. CNNs exploit the spatial structure of images through local receptive fields, shared weights and pooling operations, allowing them to learn translation-tolerant features while keeping the number of parameters tractable (Goodfellow, Bengio and Courville, 2016). The contemporary interest in CNNs is often traced to the marked improvement they produced on the ImageNet natural-image classification benchmark, where a deep architecture substantially outperformed prior approaches (Krizhevsky, Sutskever and Hinton, 2012). Subsequent architectural refinements, including very deep networks with small convolutional filters (Simonyan and Zisserman, 2015) and residual connections that ease the training of extremely deep models (He et al., 2016), established design patterns that medical imaging research adopted with relatively little modification.

Two features of the medical domain shaped how these architectures were applied. First, labelled medical datasets are typically far smaller than natural-image corpora, which encouraged the widespread use of transfer learning, where networks pre-trained on general images are fine-tuned on clinical data (Shen, Wu and Suk, 2017). Second, many diagnostic tasks require segmentation, the pixel-level delineation of anatomical or pathological regions, rather than whole-image classification. The U-Net architecture, with its symmetric encoder-decoder structure and skip connections, was proposed specifically for biomedical segmentation and has become a de facto standard in the field (Ronneberger, Fischer and Brox, 2015). The comprehensive survey by Litjens et al. (2017) documents how these building blocks were assembled across hundreds of studies, and it remains a common reference point for situating later work.

Thematic Synthesis

Applications across clinical domains

In radiology, deep learning has been applied to modalities ranging from chest radiography to computed tomography and magnetic resonance imaging. A frequently cited example is the detection of pathology on chest X-rays, where large publicly released datasets enabled the training of networks to flag conditions such as pneumonia and other thoracic abnormalities (Rajpurkar et al., 2017). Volumetric applications include the analysis of low-dose computed tomography for lung cancer screening, where end-to-end models were trained to assess whole scans (Ardila et al., 2019). Mammography has attracted particular attention because of established population screening programmes; retrospective work has evaluated deep models against radiologist reads on large screening cohorts (McKinney et al., 2020).

In pathology, the digitisation of glass slides into gigapixel whole-slide images created both an opportunity and a computational challenge. A single slide may contain billions of pixels, which forces most pipelines to divide the image into smaller tiles, classify each tile, and then aggregate the results into a slide-level decision. Studies have trained CNNs to classify tumour subtypes and predict molecular features directly from histopathology tiles (Coudray et al., 2018), and to detect the metastatic spread of breast cancer to lymph nodes, a task formalised in the widely used CAMELYON challenge and its associated benchmark comparison against pathologists (Ehteshami Bejnordi et al., 2017). Because pixel-level annotation of enormous slides is costly, weakly supervised approaches that learn from slide-level diagnostic labels alone have been proposed to make clinical-scale training feasible (Campanella et al., 2019). This design choice recurs throughout the pathology literature and illustrates a broader theme: architectural and training decisions in medical imaging are frequently driven as much by the practical economics of annotation as by considerations of raw model capacity.

Ophthalmology has proved an especially productive testbed, in part because retinal fundus photographs are standardised and abundant. Deep learning systems have been developed to detect diabetic retinopathy from fundus images (Gulshan et al., 2016), with parallel work validating similar systems across multi-ethnic populations (Ting et al., 2017). Extending beyond a single disease, De Fauw et al. (2018) reported a system using optical coherence tomography to triage a range of retinal conditions and referral decisions. Across all three domains, the recurring pattern is one of narrowly scoped, well-defined diagnostic tasks paired with comparatively large curated datasets.

Performance and benchmark studies

The evidence base for deep learning in medical imaging rests heavily on benchmark and comparison studies, and the literature is careful about how performance is framed. Rather than treating any single reported metric as definitive, several authors emphasise comparison against clinician performance on matched tasks and datasets. Esteva et al. (2017), for instance, evaluated a dermatological image classifier against board-certified specialists, and studies in ophthalmology and pathology adopted analogous designs (Gulshan et al., 2016; Ehteshami Bejnordi et al., 2017). Reviews of the clinical evidence have nonetheless cautioned that many such studies are retrospective, single-site and reported using heterogeneous metrics, which complicates cross-study comparison (Topol, 2019).

A more pointed methodological critique concerns the design and reporting quality of diagnostic accuracy studies. A systematic review by Liu et al. (2019) compared deep learning performance against health-care professionals across many imaging tasks and found that comparatively few studies used external validation or prospective designs, and that reporting standards varied considerably. The same review noted that a substantial proportion of published work was excluded from quantitative synthesis precisely because it did not report results in a comparable form, which suggests that the visible performance literature may be both selective and optimistic. This has motivated the development of reporting and trial guidelines tailored to artificial intelligence interventions, which several commentators regard as necessary for the field to mature from proof-of-concept demonstrations towards clinically trustworthy evidence (Kelly et al., 2019). Taken together, the literature therefore presents strong performance signals alongside persistent doubts about whether reported figures generalise beyond the narrow and often favourable conditions under which they were obtained.

Challenges: data, generalisation, interpretability and bias

The first cluster of challenges concerns data. Supervised deep learning is dependent on large, accurately labelled datasets, yet medical annotations require scarce expert time, and privacy, governance and interoperability constraints restrict data sharing (Willemink et al., 2020). Class imbalance, inconsistent labelling protocols and variation in imaging equipment further complicate dataset construction. These conditions have made transfer learning, data augmentation and, more recently, self-supervised and weakly supervised strategies central to the field (Campanella et al., 2019; Shen, Wu and Suk, 2017).

The second challenge, generalisation, follows closely. Models frequently perform worse when evaluated on data from institutions, scanners or populations not represented in training, a manifestation of distribution shift. A widely discussed illustration is the tendency of networks to exploit spurious correlations, or shortcut features, such as site-specific markers rather than genuine pathology; DeGrave, Janizek and Lee (2021) demonstrated this in the context of COVID-19 detection from chest radiographs, where models latched onto dataset artefacts rather than disease signs. Such findings underline why external validation is treated in the literature as a prerequisite for clinical claims.

The third challenge is interpretability. Deep networks are frequently characterised as opaque, which is problematic in a domain where clinicians and regulators require justifiable decisions. A substantial body of work has developed post-hoc explanation techniques, including gradient-based class-activation mapping that highlights image regions influencing a prediction (Selvaraju et al., 2017). However, critics argue that saliency-style explanations can be unstable and may offer a false sense of transparency, and that inherently interpretable models or rigorous validation may be preferable to post-hoc rationalisation (Rudin, 2019). Interpretability thus remains contested rather than solved.

The fourth challenge, bias and equity, cuts across the others. Because models inherit the characteristics of their training data, under-representation of particular demographic groups can produce systematically unequal performance across, for example, skin tone, age or geographic origin. Although the most prominent demonstration of algorithmic bias in health concerned a risk-prediction model rather than an imaging system, its lesson, that proxies and skewed data encode inequity, is treated as directly applicable to imaging (Obermeyer et al., 2019). The concern is compounded by the shortcut-learning findings noted above, since a model that keys on site-specific or demographic artefacts may appear accurate in aggregate while failing for particular subpopulations (DeGrave, Janizek and Lee, 2021). Reviews of clinical translation accordingly position fairness auditing, and the reporting of performance disaggregated by subgroup, alongside overall accuracy as a condition for responsible deployment (Kelly et al., 2019).

Critical Appraisal and Gaps

Read together, the literature exhibits a consistent asymmetry. The volume of studies reporting strong performance on circumscribed diagnostic tasks is large and continues to grow, yet the body of evidence demonstrating robust, prospective, multi-site clinical utility remains comparatively thin. Much of the foundational and most-cited work is retrospective and internally validated, and even landmark studies tend to isolate a single task under favourable data conditions (Litjens et al., 2017; Topol, 2019). This creates a gap between technical feasibility, which is well established, and clinical effectiveness, which is far less so.

A second gap concerns methodological consistency. The absence, until relatively recently, of agreed reporting standards means that headline metrics are often not comparable across studies, and negative or null results appear under-represented in the published record (Liu et al., 2019; Kelly et al., 2019). This weakens the field’s capacity to synthesise findings quantitatively and raises the possibility that aggregate performance is overstated.

A third gap is the uneven treatment of the four challenges. Data scarcity and, increasingly, generalisation are addressed with concrete technical strategies, whereas interpretability and bias are more often acknowledged than resolved. The dispute over whether post-hoc explanation is adequate (Selvaraju et al., 2017; Rudin, 2019) illustrates an unsettled methodological debate rather than a consolidated best practice. Similarly, fairness auditing is frequently recommended but inconsistently implemented in the primary studies reviewed.

Finally, the literature is dominated by three data-rich domains, radiology, pathology and ophthalmology, which benefit from standardised, high-volume imaging. Less is known about domains where data are scarcer or more heterogeneous, and about how models behave under the continual distribution shifts of real clinical workflows. These observations point towards prospective external validation, standardised reporting, integrated interpretability, and systematic fairness evaluation as the most pressing directions for future work.

Conclusion

The literature reviewed here describes a field that has moved with unusual speed from architectural innovation to clinically oriented demonstration. Convolutional networks, adapted through transfer learning and segmentation-specific designs, now underpin a substantial research corpus spanning radiology, pathology and ophthalmology (Litjens et al., 2017; Ronneberger, Fischer and Brox, 2015). The evidence base, anchored in benchmark and clinician-comparison studies, indicates that deep learning can perform diagnostic image-interpretation tasks to a standard that warrants serious clinical interest (Gulshan et al., 2016; Esteva et al., 2017). At the same time, the synthesis makes clear that persistent challenges of data availability, generalisation, interpretability and bias temper these results, and that methodological weaknesses in study design limit confident inference about real-world utility (Topol, 2019; Kelly et al., 2019). The overarching conclusion drawn from the literature is therefore one of qualified optimism: the technical foundations are strong, but the translation of demonstrated performance into dependable, equitable clinical practice remains an open and actively contested problem.

References

Ardila, D., Kiraly, A.P., Bharadwaj, S., Choi, B., Reicher, J.J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., Naidich, D.P. and Shetty, S. (2019) ‘End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography’, Nature Medicine, 25(6), pp. 954-961.

Campanella, G., Hanna, M.G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K.J., Brogi, E., Reuter, V.E., Klimstra, D.S. and Fuchs, T.J. (2019) ‘Clinical-grade computational pathology using weakly supervised deep learning on whole slide images’, Nature Medicine, 25(8), pp. 1301-1309.

Coudray, N., Ocampo, P.S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyo, D., Moreira, A.L., Razavian, N. and Tsirigos, A. (2018) ‘Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning’, Nature Medicine, 24(10), pp. 1559-1567.

De Fauw, J., Ledsam, J.R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D. et al. (2018) ‘Clinically applicable deep learning for diagnosis and referral in retinal disease’, Nature Medicine, 24(9), pp. 1342-1350.

DeGrave, A.J., Janizek, J.D. and Lee, S.-I. (2021) ‘AI for radiographic COVID-19 detection selects shortcuts over signal’, Nature Machine Intelligence, 3(7), pp. 610-619.

Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J.A.W.M. and the CAMELYON16 Consortium (2017) ‘Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer’, JAMA, 318(22), pp. 2199-2210.

Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M. and Thrun, S. (2017) ‘Dermatologist-level classification of skin cancer with deep neural networks’, Nature, 542(7639), pp. 115-118.

Goodfellow, I., Bengio, Y. and Courville, A. (2016) Deep Learning. Cambridge, MA: MIT Press.

Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J. et al. (2016) ‘Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs’, JAMA, 316(22), pp. 2402-2410.

He, K., Zhang, X., Ren, S. and Sun, J. (2016) ‘Deep residual learning for image recognition’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, pp. 770-778.

Kelly, C.J., Karthikesalingam, A., Sudlow, M., Hughes, C., Rees, G. and King, D. (2019) ‘Key challenges for delivering clinical impact with artificial intelligence’, BMC Medicine, 17(1), 195.

Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) ‘ImageNet classification with deep convolutional neural networks’, in Advances in Neural Information Processing Systems (NIPS) 25. Lake Tahoe, NV, pp. 1097-1105.

LeCun, Y., Bengio, Y. and Hinton, G. (2015) ‘Deep learning’, Nature, 521(7553), pp. 436-444.

Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A.W.M., van Ginneken, B. and Sanchez, C.I. (2017) ‘A survey on deep learning in medical image analysis’, Medical Image Analysis, 42, pp. 60-88.

Liu, X., Faes, L., Kale, A.U., Wagner, S.K., Fu, D.J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C. et al. (2019) ‘A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis’, The Lancet Digital Health, 1(6), pp. e271-e297.

McKinney, S.M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G.S., Darzi, A. et al. (2020) ‘International evaluation of an AI system for breast cancer screening’, Nature, 577(7788), pp. 89-94.

Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019) ‘Dissecting racial bias in an algorithm used to manage the health of populations’, Science, 366(6464), pp. 447-453.

Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., Lungren, M.P. and Ng, A.Y. (2017) ‘CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning’, arXiv preprint arXiv:1711.05225.

Ronneberger, O., Fischer, P. and Brox, T. (2015) ‘U-Net: convolutional networks for biomedical image segmentation’, in Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2015. Munich, pp. 234-241.

Rudin, C. (2019) ‘Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead’, Nature Machine Intelligence, 1(5), pp. 206-215.

Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D. and Batra, D. (2017) ‘Grad-CAM: visual explanations from deep networks via gradient-based localization’, in Proceedings of the IEEE International Conference on Computer Vision (ICCV). Venice, pp. 618-626.

Shen, D., Wu, G. and Suk, H.-I. (2017) ‘Deep learning in medical image analysis’, Annual Review of Biomedical Engineering, 19, pp. 221-248.

Simonyan, K. and Zisserman, A. (2015) ‘Very deep convolutional networks for large-scale image recognition’, in International Conference on Learning Representations (ICLR). San Diego, CA.

Ting, D.S.W., Cheung, C.Y.-L., Lim, G., Tan, G.S.W., Quang, N.D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I.Y., Lee, S.Y. et al. (2017) ‘Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes’, JAMA, 318(22), pp. 2211-2223.

Topol, E.J. (2019) ‘High-performance medicine: the convergence of human and artificial intelligence’, Nature Medicine, 25(1), pp. 44-56.

Willemink, M.J., Koszek, W.A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L.R., Summers, R.M., Rubin, D.L. and Lungren, M.P. (2020) ‘Preparing medical imaging data for machine learning’, Radiology, 295(1), pp. 4-15.