1Transdisciplinary Department of Medicine & Advanced Technology, Seoul National University Hospital, Seoul, Korea
2Department of Hospital Pathology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Korea
3Interdisciplinary Program in Bioengineering, College of Engineering, Seoul National University, Seoul, Korea
4Department of Biomedical Engineering, Asan Medical Institute of Convergence Science and Technology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
5Department of Pathology, Seoul National University Bundang Hospital, Seoul National University College of Medicine, Seongnam, Korea
6Health Innovation Big Data Center, Asan Institute of Life Science, Asan Medical Center, Seoul, Korea
7Department of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
8Department of Convergence Medicine, Asan Institute of Life Science, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
9Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, Korea
10Graduate School of AI, Korea Advanced Institute of Science and Technology, Daejeon, Korea
11Knowledge of AI Lab, NCSOFT, Seongnam, Korea
12Medical Science Research Center, Ansan Hospital, Korea University College of Medicine, Ansan, Korea
13School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Korea
14Department of Pathology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
Copyright © 2023 by the Korean Cancer Association
This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Ethical Statement
The study protocols were approved by the Institutional Review Board Committees of AMC (IRB number: 2018-0583), University of Ulsan College of Medicine, Seoul, Korea, and SNUBH (IRB number: B-1806-472-106), Seoul National University College of Medicine, Gyeonggi, Korea, which waived the need for informed patient consent.
Author Contributions
Conceived and designed the analysis: Kim S (Sungchul Kim), Kim M, Ahn S, Lee H, Yang DH, Kim N, Kim S (Sungwan Kim), Park SY, Gong G.
Collected the data: Gong G, Park SY.
Contributed data or analysis tools: Kim M, Ahn S, Lee H.
Performed the analysis: Kim YG, Song IH, Cho SY.
Wrote the paper: Kim YG, Song IH, Cho SY, Kim S (Sungchul Kim), Kim M, Ahn S, Lee H, Yang DH, Kim N, Kim S (Sungwan Kim), Kim T, Kim D, Choi J, Lee KS, Ma M, Jo M, Park SY, Gong G.
Searched literature: Kim YG, Song IH, Cho SY.
Supervision: Kim S (Sungchul Kim), Kim M, Ahn S, Lee H, Yang DH, Kim N, Kim S (Sungwan Kim), Park SY, Gong G.
Experimented with algorithms: Kim T, Kim D, Choi J, Lee KS, Ma M, Jo M.
Conflicts of Interest
Conflict of interest relevant to this article was not reported.
Training set AMC (n=236) |
Development set |
Validation set SNUBH (n=181) |
p-valuea) | ||
---|---|---|---|---|---|
AMC (n=61) | SNUBH (n=46) | ||||
Age (yr) | 50 (30–72) | 49 (28–80) | 51 (38–73) | 52 (25–87) | |
Female sex | 236 (100) | 61 (100) | 46 (100) | 181 (100) | > 0.99 |
Metastatic carcinoma | |||||
Size > 2 mm | 113 (47.9) | 30 (49.2) | 18 (39.1) | 65 (35.9) | 0.114 |
Size ≤ 2 mm | 28 (11.9) | 6 (9.8) | 8 (17.4) | 36 (19.9) | |
Absent | 95 (40.2) | 25 (41.0) | 20 (43.5) | 80 (44.2) | |
Neo-adjuvant therapy | |||||
Not received | 122 (51.7) | 30 (49.2) | 42 (91.3) | 167 (92.3) | < 0.001 |
Received | 114 (48.3) | 31 (50.8) | 4 (8.7) | 14 (7.7) | |
Histologic type | |||||
IDC | 201 (85.2) | 50 (82.0) | 46 (100) | 177 (97.8) | < 0.001b) |
ILC | 18 (7.6) | 4 (6.5) | 0 | 2 (1.1) | |
Others | 17 (7.2) | 7 (11.5) | 0 | 2 (1.1) | |
Histologic grade | |||||
1 or 2 | 188 (79.7) | 49 (80.3) | 28 (60.9) | 112 (61.9) | < 0.001 |
3 | 48 (20.3) | 12 (19.7) | 18 (39.1) | 69 (38.1) |
Values are presented as median (range) or number (%). AMC, Asan Medical Center; IDC, invasive ductal carcinoma; ILC, invasive lobular carcinoma; SNUBH, Seoul National University Bundang Hospital.
a) p-values, calculated using the chi-square test,
b) For the histologic type, a chi-square test was conducted between IDC and non-IDC.
Team | Phase 2. Validation set (SNUBH) | ||||
---|---|---|---|---|---|
ACC | TPR | TNR | PPV | NPV | |
GoldenPass | 0.845 | 0.772 | 0.938 | 0.940 | 0.765 |
MediTrain | 0.790 | 0.644 | 0.975 | 0.970 | 0.684 |
DRM | 0.724 | 0.634 | 0.838 | 0.831 | 0.644 |
Team | AUC (validation set) | |
---|---|---|
Absent vs. Tumor (including ≤ 2 mm) (Fig. 1) | Absent (including ≤ 2 mm) vs. Tumor (Fig. 2) | |
GoldenPass | 0.891 | 0.976 |
MediTrain | 0.809 | 0.823 |
DRM | 0.736 | 0.648 |
Team | Scores of major axis prediction error range | ||||
---|---|---|---|---|---|
±5% | ±10% | ±15% | ±20% | ±30% | |
GoldenPass | 0.525 | 0.552 | 0.597 | 0.613 | 0.630 |
MediTrain | 0.459 | 0.492 | 0.508 | 0.525 | 0.580 |
DRM | 0.387 | 0.475 | 0.492 | 0.503 | 0.514 |
Clinicopathologic characteristics of the patients in the AMC and SNUBH datasets (resolution [width×height] of the digital slide: 93,615×232,948 pixels [AMC] and 56,462×132,956 pixels [SNUBH])
Training set AMC (n=236) |
Development set | Validation set SNUBH (n=181) |
p-value | ||
---|---|---|---|---|---|
AMC (n=61) | SNUBH (n=46) | ||||
Age (yr) | 50 (30–72) | 49 (28–80) | 51 (38–73) | 52 (25–87) | |
Female sex | 236 (100) | 61 (100) | 46 (100) | 181 (100) | > 0.99 |
Metastatic carcinoma | |||||
Size > 2 mm | 113 (47.9) | 30 (49.2) | 18 (39.1) | 65 (35.9) | 0.114 |
Size ≤ 2 mm | 28 (11.9) | 6 (9.8) | 8 (17.4) | 36 (19.9) | |
Absent | 95 (40.2) | 25 (41.0) | 20 (43.5) | 80 (44.2) | |
Neo-adjuvant therapy | |||||
Not received | 122 (51.7) | 30 (49.2) | 42 (91.3) | 167 (92.3) | < 0.001 |
Received | 114 (48.3) | 31 (50.8) | 4 (8.7) | 14 (7.7) | |
Histologic type | |||||
IDC | 201 (85.2) | 50 (82.0) | 46 (100) | 177 (97.8) | < 0.001 |
ILC | 18 (7.6) | 4 (6.5) | 0 | 2 (1.1) | |
Others | 17 (7.2) | 7 (11.5) | 0 | 2 (1.1) | |
Histologic grade | |||||
1 or 2 | 188 (79.7) | 49 (80.3) | 28 (60.9) | 112 (61.9) | < 0.001 |
3 | 48 (20.3) | 12 (19.7) | 18 (39.1) | 69 (38.1) |
Values are presented as median (range) or number (%). AMC, Asan Medical Center; IDC, invasive ductal carcinoma; ILC, invasive lobular carcinoma; SNUBH, Seoul National University Bundang Hospital.
a)p-values, calculated using the chi-square test,
b)For the histologic type, a chi-square test was conducted between IDC and non-IDC.
Final scores of performances of classification of tumor slides and prediction of major axes
Team | Phase 1. Development set (AMC+SNUBH) | Phase 2. Validation set (SNUBH) | ||||
---|---|---|---|---|---|---|
|
| |||||
AUC of slides | Scores of major axis | Total score | AUC of slides | Scores of major axis | Total score | |
GoldenPass | 0.901 | 0.523 | 0.712 | 0.891 | 0.525 | 0.708 |
| ||||||
MediTrain | 0.838 | 0.411 | 0.624 | 0.809 | 0.459 | 0.634 |
| ||||||
DRM | 0.542 | 0.402 | 0.472 | 0.736 | 0.387 | 0.561 |
AMC, Asan Medical Center; AUC, area under the curve; DRM, DeepRunningMachine; SNUBH, Seoul National University Bundang Hospital.
Performance comparison of classification task of tumor slides
Team | Phase 2. Validation set (SNUBH) | ||||
---|---|---|---|---|---|
ACC | TPR | TNR | PPV | NPV | |
GoldenPass | 0.845 | 0.772 | 0.938 | 0.940 | 0.765 |
MediTrain | 0.790 | 0.644 | 0.975 | 0.970 | 0.684 |
DRM | 0.724 | 0.634 | 0.838 | 0.831 | 0.644 |
ACC, accuracy; DRM, DeepRunningMachine; NPV, negative predictive value; PPV, positive predictive value; SNUBH, Seoul National University Bundang Hospital; TNR, true-negative rate; TPR, true-positive rate.
Performance comparison of classifying micro-metastasis as tumor versus micro-metastasis as normal
Team | AUC (validation set) | |
---|---|---|
Absent vs. Tumor (including ≤ 2 mm) ( |
Absent (including ≤ 2 mm) vs. Tumor ( | |
GoldenPass | 0.891 | 0.976 |
MediTrain | 0.809 | 0.823 |
DRM | 0.736 | 0.648 |
AUC, area under the curve; DRM, DeepRunningMachine.
Performance comparison of determining clinicopathologic characteristics of tumors
Team | |||
---|---|---|---|
GoldenPass | MediTrain | DeepRunningMachine | |
Metastatic tumor size | |||
Absent (n=80) | |||
TPR | 0.938 | 0.975 | 0.838 |
FNR | 0.063 | 0.025 | 0.163 |
≤ 2 mm (n=36) | |||
TPR | 0.361 | 0.389 | 0.667 |
FNR | 0.639 | 0.611 | 0.333 |
> 2 mm (n=65) | |||
TPR | 1.000 | 0.785 | 0.615 |
FNR | 0.000 | 0.215 | 0.385 |
Neoadjuvant therapy | |||
Not received (n=167) | |||
TPR | 0.766 | 0.649 | 0.628 |
TNR | 0.959 | 0.986 | 0.836 |
Received (n=14) | |||
TPR | 0.857 | 0.571 | 0.714 |
TNR | 0.714 | 0.857 | 0.857 |
Histologic type | |||
IDC (n=177) | |||
TPR | 0.765 | 0.643 | 0.633 |
TNR | 0.937 | 0.975 | 0.835 |
ILC+mixed (n=4) | |||
TPR | 1.000 | 0.667 | 0.667 |
TNR | 1.000 | 1.000 | 1.000 |
Histologic grade | |||
1 or 2 (n=112) | |||
TPR | 0.794 | 0.619 | 0.619 |
TNR | 0.939 | 0.980 | 0.796 |
3 (n=69) | |||
TPR | 0.737 | 0.684 | 0.658 |
TNR | 0.935 | 0.968 | 0.903 |
FNR, false-negative rate; IDC, invasive ductal carcinoma; ILC, invasive lobular carcinoma; TNR, true-negative rate; TPR, true-positive rate.
Performance differences in major axis prediction according to variations in error range
Team | Scores of major axis prediction error range | ||||
---|---|---|---|---|---|
±5% | ±10% | ±15% | ±20% | ±30% | |
GoldenPass | 0.525 | 0.552 | 0.597 | 0.613 | 0.630 |
MediTrain | 0.459 | 0.492 | 0.508 | 0.525 | 0.580 |
DRM | 0.387 | 0.475 | 0.492 | 0.503 | 0.514 |
DRM, DeepRunningMachine.
Values are presented as median (range) or number (%). AMC, Asan Medical Center; IDC, invasive ductal carcinoma; ILC, invasive lobular carcinoma; SNUBH, Seoul National University Bundang Hospital. p-values, calculated using the chi-square test, For the histologic type, a chi-square test was conducted between IDC and non-IDC.
AMC, Asan Medical Center; AUC, area under the curve; DRM, DeepRunningMachine; SNUBH, Seoul National University Bundang Hospital.
ACC, accuracy; DRM, DeepRunningMachine; NPV, negative predictive value; PPV, positive predictive value; SNUBH, Seoul National University Bundang Hospital; TNR, true-negative rate; TPR, true-positive rate.
AUC, area under the curve; DRM, DeepRunningMachine.
FNR, false-negative rate; IDC, invasive ductal carcinoma; ILC, invasive lobular carcinoma; TNR, true-negative rate; TPR, true-positive rate.
DRM, DeepRunningMachine.