Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

Babak Ehteshami Bejnordi; Mitko Veta; Paul Johannes Van Diest; Bram Van Ginneken; Nico Karssemeijer; Geert Litjens; Jeroen A.W.M. Van Der Laak; Meyke Hermsen; Quirine F. Manson; Maschenka Balkenhol; Oscar Geessink; Nikolaos Stathonikos; Marcory C.R.F. Van Dijk; Peter Bult; Francisco Beca; Andrew H. Beck; Dayong Wang; Aditya Khosla; Rishab Gargeya; Humayun Irshad; Aoxiao Zhong; Qi Dou; Quanzheng Li; Hao Chen; Huang Jing Lin; Pheng Ann Heng; Christian Haß; Elia Bruni; Quincy Wong; Ugur Halici; Mustafa Ümit Öner; Rengul Cetin-Atalay; Matt Berseth; Vitali Khvatkov; Alexei Vylegzhanin; Oren Kraus; Muhammad Shaban; Nasir Rajpoot; Ruqayya Awan; Korsuk Sirinukunwattana; Talha Qaiser; Yee Wah Tsang; David Tellez; Jonas Annuscheit; Peter Hufnagl; Mira Valkonen; Kimmo Kartasalo; Leena Latonen; Pekka Ruusuvuori; Kaisa Liimatainen; Shadi Albarqouni; Bharti Mungal; Ami George; Stefanie Demirci; Nassir Navab; Seiryo Watanabe; Shigeto Seno; Yoichi Takenaka; Hideo Matsuda; Hady Ahmady Phoulady; Vassili Kovalev; Alexander Kalinovsky; Vitali Liauchuk; Gloria Bueno; M. Milagro Fernandez-Carrobles; Ismael Serrano; Oscar Deniz; Daniel Racoceanu; Rui Venâncio

Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen A.W.M. Van Der Laak, Meyke Hermsen, Quirine F. Manson, Maschenka Balkenhol, Oscar Geessink, Nikolaos Stathonikos, Marcory C.R.F. Van Dijk, Peter Bult, Francisco Beca, Andrew H. Beck, Dayong Wang, Aditya Khosla, Rishab Gargeya, Humayun IrshadAoxiao Zhong, Qi Dou, Quanzheng Li, Hao Chen, Huang Jing Lin, Pheng Ann Heng, Christian Haß, Elia Bruni, Quincy Wong, Ugur Halici, Mustafa Ümit Öner, Rengul Cetin-Atalay, Matt Berseth, Vitali Khvatkov, Alexei Vylegzhanin, Oren Kraus, Muhammad Shaban, Nasir Rajpoot, Ruqayya Awan, Korsuk Sirinukunwattana, Talha Qaiser, Yee Wah Tsang, David Tellez, Jonas Annuscheit, Peter Hufnagl, Mira Valkonen, Kimmo Kartasalo, Leena Latonen, Pekka Ruusuvuori, Kaisa Liimatainen, Shadi Albarqouni, Bharti Mungal, Ami George, Stefanie Demirci, Nassir Navab, Seiryo Watanabe, Shigeto Seno, Yoichi Takenaka, Hideo Matsuda, Hady Ahmady Phoulady, Vassili Kovalev, Alexander Kalinovsky, Vitali Liauchuk, Gloria Bueno, M. Milagro Fernandez-Carrobles, Ismael Serrano, Oscar Deniz, Daniel Racoceanu, Rui Venâncio

Pontificia Universidad Católica del Perú

Research output: Contribution to journal › Article › peer-review

2243 Scopus citations

Abstract

IMPORTANCE: Application of deep learning algorithms to whole-slide pathology imagescan potentially improve diagnostic accuracy and efficiency. OBJECTIVE: Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. DESIGN, SETTING, AND PARTICIPANTS: Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). EXPOSURES: Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. MAIN OUTCOMES AND MEASURES: The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. RESULTS: The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P

Original language	Spanish
Pages (from-to)	2199-2210
Number of pages	12
Journal	JAMA - Journal of the American Medical Association
Volume	318
State	Published - 12 Dec 2017

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Cite this

Bejnordi, B. E., Veta, M., Van Diest, P. J., Van Ginneken, B., Karssemeijer, N., Litjens, G., Van Der Laak, J. A. W. M., Hermsen, M., Manson, Q. F., Balkenhol, M., Geessink, O., Stathonikos, N., Van Dijk, M. C. R. F., Bult, P., Beca, F., Beck, A. H., Wang, D., Khosla, A., Gargeya, R., ... Venâncio, R. (2017). Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA - Journal of the American Medical Association, 318, 2199-2210.

@article{ac7d724455b94215a044b5284a0573c6,

title = "Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer",

abstract = "IMPORTANCE: Application of deep learning algorithms to whole-slide pathology imagescan potentially improve diagnostic accuracy and efficiency. OBJECTIVE: Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. DESIGN, SETTING, AND PARTICIPANTS: Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). EXPOSURES: Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. MAIN OUTCOMES AND MEASURES: The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. RESULTS: The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P",

author = "Bejnordi, {Babak Ehteshami} and Mitko Veta and {Van Diest}, {Paul Johannes} and {Van Ginneken}, Bram and Nico Karssemeijer and Geert Litjens and {Van Der Laak}, {Jeroen A.W.M.} and Meyke Hermsen and Manson, {Quirine F.} and Maschenka Balkenhol and Oscar Geessink and Nikolaos Stathonikos and {Van Dijk}, {Marcory C.R.F.} and Peter Bult and Francisco Beca and Beck, {Andrew H.} and Dayong Wang and Aditya Khosla and Rishab Gargeya and Humayun Irshad and Aoxiao Zhong and Qi Dou and Quanzheng Li and Hao Chen and Lin, {Huang Jing} and Heng, {Pheng Ann} and Christian Ha{\ss} and Elia Bruni and Quincy Wong and Ugur Halici and {\"O}ner, {Mustafa {\"U}mit} and Rengul Cetin-Atalay and Matt Berseth and Vitali Khvatkov and Alexei Vylegzhanin and Oren Kraus and Muhammad Shaban and Nasir Rajpoot and Ruqayya Awan and Korsuk Sirinukunwattana and Talha Qaiser and Tsang, {Yee Wah} and David Tellez and Jonas Annuscheit and Peter Hufnagl and Mira Valkonen and Kimmo Kartasalo and Leena Latonen and Pekka Ruusuvuori and Kaisa Liimatainen and Shadi Albarqouni and Bharti Mungal and Ami George and Stefanie Demirci and Nassir Navab and Seiryo Watanabe and Shigeto Seno and Yoichi Takenaka and Hideo Matsuda and Phoulady, {Hady Ahmady} and Vassili Kovalev and Alexander Kalinovsky and Vitali Liauchuk and Gloria Bueno and Fernandez-Carrobles, {M. Milagro} and Ismael Serrano and Oscar Deniz and Daniel Racoceanu and Rui Ven{\^a}ncio",

year = "2017",

month = dec,

day = "12",

language = "Espa{\~n}ol",

volume = "318",

pages = "2199--2210",

journal = "JAMA - Journal of the American Medical Association",

}

Bejnordi, BE, Veta, M, Van Diest, PJ, Van Ginneken, B, Karssemeijer, N, Litjens, G, Van Der Laak, JAWM, Hermsen, M, Manson, QF, Balkenhol, M, Geessink, O, Stathonikos, N, Van Dijk, MCRF, Bult, P, Beca, F, Beck, AH, Wang, D, Khosla, A, Gargeya, R, Irshad, H, Zhong, A, Dou, Q, Li, Q, Chen, H, Lin, HJ, Heng, PA, Haß, C, Bruni, E, Wong, Q, Halici, U, Öner, MÜ, Cetin-Atalay, R, Berseth, M, Khvatkov, V, Vylegzhanin, A, Kraus, O, Shaban, M, Rajpoot, N, Awan, R, Sirinukunwattana, K, Qaiser, T, Tsang, YW, Tellez, D, Annuscheit, J, Hufnagl, P, Valkonen, M, Kartasalo, K, Latonen, L, Ruusuvuori, P, Liimatainen, K, Albarqouni, S, Mungal, B, George, A, Demirci, S, Navab, N, Watanabe, S, Seno, S, Takenaka, Y, Matsuda, H, Phoulady, HA, Kovalev, V, Kalinovsky, A, Liauchuk, V, Bueno, G, Fernandez-Carrobles, MM, Serrano, I, Deniz, O, Racoceanu, D & Venâncio, R 2017, 'Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer', JAMA - Journal of the American Medical Association, vol. 318, pp. 2199-2210.

TY - JOUR

T1 - Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

AU - Bejnordi, Babak Ehteshami

AU - Veta, Mitko

AU - Van Diest, Paul Johannes

AU - Van Ginneken, Bram

AU - Karssemeijer, Nico

AU - Litjens, Geert

AU - Van Der Laak, Jeroen A.W.M.

AU - Hermsen, Meyke

AU - Manson, Quirine F.

AU - Balkenhol, Maschenka

AU - Geessink, Oscar

AU - Stathonikos, Nikolaos

AU - Van Dijk, Marcory C.R.F.

AU - Bult, Peter

AU - Beca, Francisco

AU - Beck, Andrew H.

AU - Wang, Dayong

AU - Khosla, Aditya

AU - Gargeya, Rishab

AU - Irshad, Humayun

AU - Zhong, Aoxiao

AU - Dou, Qi

AU - Li, Quanzheng

AU - Chen, Hao

AU - Lin, Huang Jing

AU - Heng, Pheng Ann

AU - Haß, Christian

AU - Bruni, Elia

AU - Wong, Quincy

AU - Halici, Ugur

AU - Öner, Mustafa Ümit

AU - Cetin-Atalay, Rengul

AU - Berseth, Matt

AU - Khvatkov, Vitali

AU - Vylegzhanin, Alexei

AU - Kraus, Oren

AU - Shaban, Muhammad

AU - Rajpoot, Nasir

AU - Awan, Ruqayya

AU - Sirinukunwattana, Korsuk

AU - Qaiser, Talha

AU - Tsang, Yee Wah

AU - Tellez, David

AU - Annuscheit, Jonas

AU - Hufnagl, Peter

AU - Valkonen, Mira

AU - Kartasalo, Kimmo

AU - Latonen, Leena

AU - Ruusuvuori, Pekka

AU - Liimatainen, Kaisa

AU - Albarqouni, Shadi

AU - Mungal, Bharti

AU - George, Ami

AU - Demirci, Stefanie

AU - Navab, Nassir

AU - Watanabe, Seiryo

AU - Seno, Shigeto

AU - Takenaka, Yoichi

AU - Matsuda, Hideo

AU - Phoulady, Hady Ahmady

AU - Kovalev, Vassili

AU - Kalinovsky, Alexander

AU - Liauchuk, Vitali

AU - Bueno, Gloria

AU - Fernandez-Carrobles, M. Milagro

AU - Serrano, Ismael

AU - Deniz, Oscar

AU - Racoceanu, Daniel

AU - Venâncio, Rui

PY - 2017/12/12

Y1 - 2017/12/12

N2 - IMPORTANCE: Application of deep learning algorithms to whole-slide pathology imagescan potentially improve diagnostic accuracy and efficiency. OBJECTIVE: Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. DESIGN, SETTING, AND PARTICIPANTS: Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). EXPOSURES: Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. MAIN OUTCOMES AND MEASURES: The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. RESULTS: The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P

AB - IMPORTANCE: Application of deep learning algorithms to whole-slide pathology imagescan potentially improve diagnostic accuracy and efficiency. OBJECTIVE: Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. DESIGN, SETTING, AND PARTICIPANTS: Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). EXPOSURES: Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. MAIN OUTCOMES AND MEASURES: The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. RESULTS: The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P

M3 - Artículo

VL - 318

SP - 2199

EP - 2210

JO - JAMA - Journal of the American Medical Association

JF - JAMA - Journal of the American Medical Association

ER -