Why is the winner the best?

Matthias Eisenmann; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Fabian Isensee; Tim J. Adler; Sharib Ali; Vincent Andrearczyk; Marc Aubreville; Ujjwal Baid; Spyridon Bakas; Niranjan Balu; Sophia Bano; Jorge Bernal; Sebastian Bodenstedt; Alessandro Casella; Veronika Cheplygina; Marie Daum; Marleen de Bruijne; Adrien Depeursinge; Reuben Dorent; Jan Egger; David G. Ellis; Sandy Engelhardt; Melanie Ganz; Noha Ghatwary; Gabriel Girard; Patrick Godau; Anubha Gupta; Lasse Hansen; Kanako Harada; Mattias Heinrich; Nicholas Heller; Alessa Hering; Arnaud Huaulmé; Pierre Jannin; Ali Emre Kavur; Oldřich Kodym; Michal Kozubek; Jianning Li; Hongwei Li; Jun Ma; Carlos Martín-Isla; Bjoern Menze; Alison Noble; Valentin Oreiller; Nicolas Padoy; Sarthak Pati; Kelly Payette; Tim Rädsch; Jonathan Rafael-Patiño; Vivek Singh Bawa; Stefanie Speidel; Carole H. Sudre; Kimberlin van Wijnen; Martin Wagner; Donglai Wei; Amine Yamlahi; Moi Hoon Yap; Chun Yuan; Maximilian Zenk; Aneeq Zia; David Zimmerer; Dogu Baran Aydogan; Binod Bhattarai; Louise Bloch; Raphael Brüngel; Jihoon Cho; Chanyeol Choi; Qi Dou; Ivan Ezhov; Christoph M. Friedrich; Clifton Fuller; Rebati Raman Gaire; Adrian Galdran; Álvaro García Faura; Maria Grammatikopoulou; SeulGi Hong; Mostafa Jahanifar; Ikbeom Jang; Abdolrahim Kadkhodamohammadi; Inha Kang; Florian Kofler; Satoshi Kondo; Hugo Kuijf; Mingxing Li; Minh Huan Luu; Tomaž Martinčič; Pedro Morais; Mohamed A. Naser; Bruno Oliveira; Subeen Pang; Jinah Park; Sung-Hong Park; Szymon Płotka; Elodie Puybareau; Nasir Rajpoot; Kanghyun Ryu; Numan Saeed; Adam Shephard; Pengcheng Shi; Dejan Štepec; Ronast Subedi; Guillaume Tochon; Helena R. Torres; Helene Urien; João L. Vilaça; Kareem Abdul Wahid; Haojie Wang; Jiacheng Wang; Liansheng Wang; Xiyue Wang; Benedikt Wiestler; Marek Wodzinski; Fangfang Xia; Juanying Xie; Zhiwei Xiong; Sen Yang; Yanwu Yang; Zixuan Zhao; Klaus Maier-Hein; Paul F. Jäger; Annette Kopp-Schneider; Lena Maier-Hein

Why is the winner the best?

Matthias Eisenmann, Annika Reinke, Vivienn Weru, Minu Dietlinde Tizabi, Fabian Isensee, Tim J. Adler, Sharib Ali, Vincent Andrearczyk, Marc Aubreville, Ujjwal Baid, Spyridon Bakas, Niranjan Balu, Sophia Bano, Jorge Bernal, Sebastian Bodenstedt, Alessandro Casella, Veronika Cheplygina, Marie Daum, Marleen de Bruijne, Adrien DepeursingeReuben Dorent, Jan Egger, David G. Ellis, Sandy Engelhardt, Melanie Ganz, Noha Ghatwary, Gabriel Girard, Patrick Godau, Anubha Gupta, Lasse Hansen, Kanako Harada, Mattias Heinrich, Nicholas Heller, Alessa Hering, Arnaud Huaulmé, Pierre Jannin, Ali Emre Kavur, Oldřich Kodym, Michal Kozubek, Jianning Li, Hongwei Li, Jun Ma, Carlos Martín-Isla, Bjoern Menze, Alison Noble, Valentin Oreiller, Nicolas Padoy, Sarthak Pati, Kelly Payette, Tim Rädsch, Jonathan Rafael-Patiño, Vivek Singh Bawa, Stefanie Speidel, Carole H. Sudre, Kimberlin van Wijnen, Martin Wagner, Donglai Wei, Amine Yamlahi, Moi Hoon Yap, Chun Yuan, Maximilian Zenk, Aneeq Zia, David Zimmerer, Dogu Baran Aydogan, Binod Bhattarai, Louise Bloch, Raphael Brüngel, Jihoon Cho, Chanyeol Choi, Qi Dou, Ivan Ezhov, Christoph M. Friedrich, Clifton Fuller, Rebati Raman Gaire, Adrian Galdran, Álvaro García Faura, Maria Grammatikopoulou, SeulGi Hong, Mostafa Jahanifar, Ikbeom Jang, Abdolrahim Kadkhodamohammadi, Inha Kang, Florian Kofler, Satoshi Kondo, Hugo Kuijf, Mingxing Li, Minh Huan Luu, Tomaž Martinčič, Pedro Morais, Mohamed A. Naser, Bruno Oliveira, Subeen Pang, Jinah Park, Sung-Hong Park, Szymon Płotka, Elodie Puybareau, Nasir Rajpoot, Kanghyun Ryu, Numan Saeed, Adam Shephard, Pengcheng Shi, Dejan Štepec, Ronast Subedi, Guillaume Tochon, Helena R. Torres, Helene Urien, João L. Vilaça, Kareem Abdul Wahid, Haojie Wang, Jiacheng Wang, Liansheng Wang, Xiyue Wang, Benedikt Wiestler, Marek Wodzinski, Fangfang Xia, Juanying Xie, Zhiwei Xiong, Sen Yang, Yanwu Yang, Zixuan Zhao, Klaus Maier-Hein, Paul F. Jäger, Annette Kopp-Schneider, Lena Maier-Hein

Research output: Working paper › Preprint

5 Downloads (Pure)

Abstract

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

Original language	Undefined/Unknown
Publication status	Published - 30 Mar 2023
Externally published	Yes

Keywords

cs.CV
cs.LG

Access to Document

2303.17719v1

Cite this

@techreport{8e477d5a74954d5ca0ca14bbeb044727,

title = "Why is the winner the best?",

abstract = "International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The {"}typical{"} lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.",

keywords = "cs.CV, cs.LG",

author = "Matthias Eisenmann and Annika Reinke and Vivienn Weru and Tizabi, {Minu Dietlinde} and Fabian Isensee and Adler, {Tim J.} and Sharib Ali and Vincent Andrearczyk and Marc Aubreville and Ujjwal Baid and Spyridon Bakas and Niranjan Balu and Sophia Bano and Jorge Bernal and Sebastian Bodenstedt and Alessandro Casella and Veronika Cheplygina and Marie Daum and {de Bruijne}, Marleen and Adrien Depeursinge and Reuben Dorent and Jan Egger and Ellis, {David G.} and Sandy Engelhardt and Melanie Ganz and Noha Ghatwary and Gabriel Girard and Patrick Godau and Anubha Gupta and Lasse Hansen and Kanako Harada and Mattias Heinrich and Nicholas Heller and Alessa Hering and Arnaud Huaulm{\'e} and Pierre Jannin and Kavur, {Ali Emre} and Old{\v r}ich Kodym and Michal Kozubek and Jianning Li and Hongwei Li and Jun Ma and Carlos Mart{\'i}n-Isla and Bjoern Menze and Alison Noble and Valentin Oreiller and Nicolas Padoy and Sarthak Pati and Kelly Payette and Tim R{\"a}dsch and Jonathan Rafael-Pati{\~n}o and Bawa, {Vivek Singh} and Stefanie Speidel and Sudre, {Carole H.} and {van Wijnen}, Kimberlin and Martin Wagner and Donglai Wei and Amine Yamlahi and Yap, {Moi Hoon} and Chun Yuan and Maximilian Zenk and Aneeq Zia and David Zimmerer and Aydogan, {Dogu Baran} and Binod Bhattarai and Louise Bloch and Raphael Br{\"u}ngel and Jihoon Cho and Chanyeol Choi and Qi Dou and Ivan Ezhov and Friedrich, {Christoph M.} and Clifton Fuller and Gaire, {Rebati Raman} and Adrian Galdran and Faura, {{\'A}lvaro Garc{\'i}a} and Maria Grammatikopoulou and SeulGi Hong and Mostafa Jahanifar and Ikbeom Jang and Abdolrahim Kadkhodamohammadi and Inha Kang and Florian Kofler and Satoshi Kondo and Hugo Kuijf and Mingxing Li and Luu, {Minh Huan} and Toma{\v z} Martin{\v c}i{\v c} and Pedro Morais and Naser, {Mohamed A.} and Bruno Oliveira and Subeen Pang and Jinah Park and Sung-Hong Park and Szymon P{\l}otka and Elodie Puybareau and Nasir Rajpoot and Kanghyun Ryu and Numan Saeed and Adam Shephard and Pengcheng Shi and Dejan {\v S}tepec and Ronast Subedi and Guillaume Tochon and Torres, {Helena R.} and Helene Urien and Vila{\c c}a, {Jo{\~a}o L.} and Wahid, {Kareem Abdul} and Haojie Wang and Jiacheng Wang and Liansheng Wang and Xiyue Wang and Benedikt Wiestler and Marek Wodzinski and Fangfang Xia and Juanying Xie and Zhiwei Xiong and Sen Yang and Yanwu Yang and Zixuan Zhao and Klaus Maier-Hein and J{\"a}ger, {Paul F.} and Annette Kopp-Schneider and Lena Maier-Hein",

note = "accepted to CVPR 2023",

year = "2023",

month = mar,

day = "30",

language = "Undefined/Unknown",

type = "WorkingPaper",

}

Eisenmann, M, Reinke, A, Weru, V, Tizabi, MD, Isensee, F, Adler, TJ, Ali, S, Andrearczyk, V, Aubreville, M, Baid, U, Bakas, S, Balu, N, Bano, S, Bernal, J, Bodenstedt, S, Casella, A, Cheplygina, V, Daum, M, de Bruijne, M, Depeursinge, A, Dorent, R, Egger, J, Ellis, DG, Engelhardt, S, Ganz, M, Ghatwary, N, Girard, G, Godau, P, Gupta, A, Hansen, L, Harada, K, Heinrich, M, Heller, N, Hering, A, Huaulmé, A, Jannin, P, Kavur, AE, Kodym, O, Kozubek, M, Li, J, Li, H, Ma, J, Martín-Isla, C, Menze, B, Noble, A, Oreiller, V, Padoy, N, Pati, S, Payette, K, Rädsch, T, Rafael-Patiño, J, Bawa, VS, Speidel, S, Sudre, CH, van Wijnen, K, Wagner, M, Wei, D, Yamlahi, A, Yap, MH, Yuan, C, Zenk, M, Zia, A, Zimmerer, D, Aydogan, DB, Bhattarai, B, Bloch, L, Brüngel, R, Cho, J, Choi, C, Dou, Q, Ezhov, I, Friedrich, CM, Fuller, C, Gaire, RR, Galdran, A, Faura, ÁG, Grammatikopoulou, M, Hong, S, Jahanifar, M, Jang, I, Kadkhodamohammadi, A, Kang, I, Kofler, F, Kondo, S, Kuijf, H, Li, M, Luu, MH, Martinčič, T, Morais, P, Naser, MA, Oliveira, B, Pang, S, Park, J, Park, S-H, Płotka, S, Puybareau, E, Rajpoot, N, Ryu, K, Saeed, N, Shephard, A, Shi, P, Štepec, D, Subedi, R, Tochon, G, Torres, HR, Urien, H, Vilaça, JL, Wahid, KA, Wang, H, Wang, J, Wang, L, Wang, X, Wiestler, B, Wodzinski, M, Xia, F, Xie, J, Xiong, Z, Yang, S, Yang, Y, Zhao, Z, Maier-Hein, K, Jäger, PF, Kopp-Schneider, A & Maier-Hein, L 2023 'Why is the winner the best?'.