Challenges drive the state-of-the-art of automated medical image analysis. The quantity of public training data that they provide can limit the performance of their solutions. Public access to the training methodology for these solutions remains absent. This study implements the Type Three (T3) challenge format, which allows for training solutions on private data and guarantees reusable training methodologies. With T3, challenge organizers train a codebase provided by the participants on sequestered training data. T3 was implemented in the STOIC2021 challenge, with the goal of predicting from a computed tomography (CT) scan whether subjects had a severe COVID-19 infection, defined as intubation or death within one month. STOIC2021 consisted of a Qualification phase, where participants developed challenge solutions using 2000 publicly available CT scans, and a Final phase, where participants submitted their training methodologies with which solutions were trained on CT scans of 9724 subjects. The organizers successfully trained six of the eight Final phase submissions. The submitted codebases for training and running inference were released publicly. The winning solution obtained an area under the receiver operating characteristic curve for discerning between severe and non-severe COVID-19 of 0.815. The Final phase solutions of all finalists improved upon their Qualification phase solutions.
Boulogne, LH, Lorenz, J, Kienzle, D, Schon, R, Ludwig, K, Lienhart, R, Jegou, S, Li, G, Chen, C, Wang, Q, Shi, D, Maniparambil, M, Muller, D, Mertes, S, Schroter, N, Hellmann, F, Elia, M, Dirks, I, Bossa, MN, Berenguer, AD, Mukherjee, T, Vandemeulebroucke, J, Sahli, H, Deligiannis, N, Gonidakis, P, Huynh, ND, Razzak, I, Bouadjenek, R, Verdicchio, M, Borrelli, P, Aiello, M, Meakin, JA, Lemm, A, Russ, C, Ionasec, R, Paragios, N, Ginneken, BV & Dubois, M-PR 2024, 'The STOIC2021 COVID-19 AI challenge: applying reusable training methodologies to private data', Medical Image Analysis, vol. 97, 103230. https://doi.org/10.1016/j.media.2024.103230, https://doi.org/10.48550/arXiv.2306.10484
Boulogne, L. H., Lorenz, J., Kienzle, D., Schon, R., Ludwig, K., Lienhart, R., Jegou, S., Li, G., Chen, C., Wang, Q., Shi, D., Maniparambil, M., Muller, D., Mertes, S., Schroter, N., Hellmann, F., Elia, M., Dirks, I., Bossa, M. N., ... Dubois, M.-P. R. (2024). The STOIC2021 COVID-19 AI challenge: applying reusable training methodologies to private data. Medical Image Analysis, 97, Article 103230. https://doi.org/10.1016/j.media.2024.103230, https://doi.org/10.48550/arXiv.2306.10484
@article{868c3344ac2d4398b6be51ae4e0e2fea,
title = "The STOIC2021 COVID-19 AI challenge: applying reusable training methodologies to private data",
abstract = " Challenges drive the state-of-the-art of automated medical image analysis. The quantity of public training data that they provide can limit the performance of their solutions. Public access to the training methodology for these solutions remains absent. This study implements the Type Three (T3) challenge format, which allows for training solutions on private data and guarantees reusable training methodologies. With T3, challenge organizers train a codebase provided by the participants on sequestered training data. T3 was implemented in the STOIC2021 challenge, with the goal of predicting from a computed tomography (CT) scan whether subjects had a severe COVID-19 infection, defined as intubation or death within one month. STOIC2021 consisted of a Qualification phase, where participants developed challenge solutions using 2000 publicly available CT scans, and a Final phase, where participants submitted their training methodologies with which solutions were trained on CT scans of 9724 subjects. The organizers successfully trained six of the eight Final phase submissions. The submitted codebases for training and running inference were released publicly. The winning solution obtained an area under the receiver operating characteristic curve for discerning between severe and non-severe COVID-19 of 0.815. The Final phase solutions of all finalists improved upon their Qualification phase solutions. ",
author = "Boulogne, {Luuk H.} and Julian Lorenz and Daniel Kienzle and Robin Schon and Katja Ludwig and Rainer Lienhart and Simon Jegou and Guang Li and Cong Chen and Qi Wang and Derik Shi and Mayug Maniparambil and Dominik Muller and Silvan Mertes and Niklas Schroter and Fabio Hellmann and Miriam Elia and Ine Dirks and Bossa, {Matias Nicolas} and Berenguer, {Abel Diaz} and Tanmoy Mukherjee and Jef Vandemeulebroucke and Hichem Sahli and Nikos Deligiannis and Panagiotis Gonidakis and Huynh, {Ngoc Dung} and Imran Razzak and Reda Bouadjenek and Mario Verdicchio and Pasquale Borrelli and Marco Aiello and Meakin, {James A.} and Alexander Lemm and Christoph Russ and Razvan Ionasec and Nikos Paragios and Ginneken, {Bram van} and Dubois, {Marie-Pierre Revel}",
note = "Funding Information: The European Regional Development Fund had no role in the study design, data collection, data analysis, data interpretation, or writing of the manuscript. Amazon Web Services funded algorithm evaluation, algorithm training for the Final phase, and prizes to the best performing teams. This study was endorsed by The Medical Image Computing and Computer Assisted Intervention (MICCAI) Society. The STOIC study ( Revel et al., 2021 ) was sponsored by Assistance Publique H\u00F4pitaux de Paris and was funded by Fondation APHP pour la Recherche , Guerbet, Innothera, Fondation CentraleSup\u00E9lec. For the STOIC study, General Electric Healthcare provided a 3D image visualization web application and Orange Healthcare a data repository. Funding Information: The European Regional Development Fund had no role in the study design, data collection, data analysis, data interpretation, or writing of the manuscript. Amazon Web Services funded algorithm evaluation, algorithm training for the Final phase, and prizes to the best performing teams. This study was endorsed by The Medical Image Computing and Computer Assisted Intervention (MICCAI) Society. The STOIC study (Revel et al. 2021) was sponsored by Assistance Publique H\u00F4pitaux de Paris and was funded by Fondation APHP pour la Recherche, Guerbet, Innothera, Fondation CentraleSup\u00E9lec. For the STOIC study, General Electric Healthcare provided a 3D image visualization web application and Orange Healthcare a data repository. The European Regional Development Fund had no role in the study design, data collection, data analysis, data interpretation, or writing of the manuscript. Amazon Web Services funded algorithm evaluation, algorithm training for the Final phase, and prizes to the best performing teams. Publisher Copyright: {\textcopyright} 2024 The Author(s)",
year = "2024",
month = oct,
doi = "10.1016/j.media.2024.103230",
language = "English",
volume = "97",
journal = "Medical Image Analysis",
issn = "1361-8415",
publisher = "Elsevier",
}