Repeat it without me: Crowdsourcing the T1 mapping common ground via the ISMRM reproducibility challenge

Mathieu Boudreau; Agah Karakuzu; Julien Cohen-Adad; Ecem Bozkurt; Madeline Carr; Marco Castellaro; Luis Concha; Mariya Doneva; Seraina A Dual; Alex Ensworth; Alexandru Foias; Véronique Fortier; Refaat E Gabr; Guillaume Gilbert; Carri K Glide-Hurst; Matthew Grech-Sollars; Siyuan Hu; Oscar Jalnefjord; Jorge Jovicich; Kübra Keskin; Peter Koken; Anastasia Kolokotronis; Simran Kukran; Nam G Lee; Ives R Levesque; Bochao Li; Dan Ma; Burkhard Mädler; Nyasha G Maforo; Jamie Near; Erick Pasaye; Alonso Ramirez-Manzanares; Ben Statton; Christian Stehning; Stefano Tambalo; Ye Tian; Chenyang Wang; Kilian Weiss; Niloufar Zakariaei; Shuo Zhang; Ziwei Zhao; Nikola Stikov; ISMRM Reproducible Research Study Group and the ISMRM Quantitative MR Study Group

doi:10.1002/mrm.30111

Repeat it without me: Crowdsourcing the T₁ mapping common ground via the ISMRM reproducibility challenge

Magn Reson Med. 2024 May 10. doi: 10.1002/mrm.30111. Online ahead of print.

Authors

Mathieu Boudreau^{1

2}, Agah Karakuzu¹, Julien Cohen-Adad^{1

2

3

4

5}, Ecem Bozkurt⁶, Madeline Carr^{7

8}, Marco Castellaro⁹, Luis Concha¹⁰, Mariya Doneva¹¹, Seraina A Dual¹², Alex Ensworth^{13

14}, Alexandru Foias¹, Véronique Fortier^{15

16}, Refaat E Gabr¹⁷, Guillaume Gilbert¹⁸, Carri K Glide-Hurst¹⁹, Matthew Grech-Sollars^{20

21}, Siyuan Hu²², Oscar Jalnefjord^{23

24}, Jorge Jovicich²⁵, Kübra Keskin⁶, Peter Koken¹¹, Anastasia Kolokotronis^{13

26}, Simran Kukran^{27

28}, Nam G Lee⁶, Ives R Levesque^{13

29}, Bochao Li⁶, Dan Ma²², Burkhard Mädler³⁰, Nyasha G Maforo^{31

32}, Jamie Near^{33

34}, Erick Pasaye¹⁰, Alonso Ramirez-Manzanares³⁵, Ben Statton³⁶, Christian Stehning³⁰, Stefano Tambalo²⁵, Ye Tian⁶, Chenyang Wang³⁷, Kilian Weiss³⁰, Niloufar Zakariaei³⁸, Shuo Zhang³⁰, Ziwei Zhao⁶, Nikola Stikov^{1

2

39}; ISMRM Reproducible Research Study Group and the ISMRM Quantitative MR Study Group

Affiliations

¹ NeuroPoly Lab, Polytechnique Montréal, Montréal, Quebec, Canada.
² Montreal Heart Institute, Montréal, Quebec, Canada.
³ Unité de Neuroimagerie Fonctionnelle, Centre de Recherche de l'Institut Universitaire de Gériatrie de Montréal, Montréal, Quebec, Canada.
⁴ Mila-Quebec AI Institute, Montréal, Québec, Canada.
⁵ Centre de Recherche du CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada.
⁶ Magnetic Resonance Engineering Laboratory, University of Southern California, Los Angeles, California, USA.
⁷ Medical Physics, Ingham Institute for Applied Medical Research, Liverpool, Australia.
⁸ Department of Medical Physics, Liverpool and Macarthur Cancer Therapy Centers, Liverpool, Australia.
⁹ Department of Information Engineering, University of Padova, Padova, Italy.
¹⁰ Institute of Neurobiology, Universidad Nacional Autónoma de México Campus Juriquilla, Querétaro, Mexico.
¹¹ Philips Research Hamburg, Hamburg, Germany.
¹² Department of Radiology, Stanford University, Stanford, California, USA.
¹³ Medical Physics Unit, McGill University, Montréal, Québec, Canada.
¹⁴ University of British Columbia, Vancouver, British Columbia, Canada.
¹⁵ Department of Medical Imaging, McGill University Health Center, Montréal, Québec, Canada.
¹⁶ Department of Radiology, McGill University, Montréal, Québec, Canada.
¹⁷ Department of Diagnostic and Interventional Imaging, University of Texas Health Science Center at Houston, McGovern Medical School, Houston, Texas, USA.
¹⁸ MR Clinical Science, Philips Canada, Mississauga, Ontario, Canada.
¹⁹ Department of Human Oncology, University of Wisconsin-Madison, Madison, Wisconsin, USA.
²⁰ Center for Medical Image Computing, Department of Computer Science, University College London, London, UK.
²¹ Lysholm Department of Neuroradiology, National Hospital for Neurology and Neurosurgery, University College London Hospitals NHS Foundation Trust, London, UK.
²² Department of Biomedical Engineering, Case Western Reserve University, Cleveland, Ohio, USA.
²³ Department of Medical Radiation Sciences, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.
²⁴ Department of Medical Physics and Biomedical Engineering, Sahlgrenska University Hospital, Gothenburg, Sweden.
²⁵ Center for Mind/Brain Sciences, University of Trento, Trento, Italy.
²⁶ Hopital Maisonneuve-Rosemont, Montréal, Québec, Canada.
²⁷ Bioengineering, Imperial College London, London, UK.
²⁸ Radiotherapy and Imaging, Institute of Cancer Research, Imperial College London, London, UK.
²⁹ Research Institute of the McGill University Health Center, Montréal, Québec, Canada.
³⁰ Clinical Science, Philips Healthcare, Hamburg, Germany.
³¹ Department of Radiological Sciences, University of California Los Angeles, Los Angeles, California, USA.
³² Physics and Biology in Medicine IDP, University of California Los Angeles, Los Angeles, California, USA.
³³ Douglas Brain Imaging Center, Montréal, Québec, Canada.
³⁴ Sunnybrook Research Institute, Toronto, Ontario, Canada.
³⁵ Computer Science Department, Centro de Investigación en Matemáticas, A.C., Guanajuato, Mexico.
³⁶ Medical Research Council, London Institute of Medical Sciences, Imperial College London, London, UK.
³⁷ Department of Radiation Oncology-CNS Service, The University of Texas MD Anderson Cancer Center, Houston, Texas, USA.
³⁸ Department of Biomedical Engineering, University of British Columbia, Vancouver, British Columbia, Canada.
³⁹ Center for Advanced Interdisciplinary Research, Ss. Cyril and Methodius University, Skopje, North Macedonia.

PMID: 38730562
DOI: 10.1002/mrm.30111

Abstract

Purpose: T₁ mapping is a widely used quantitative MRI technique, but its tissue-specific values remain inconsistent across protocols, sites, and vendors. The ISMRM Reproducible Research and Quantitative MR study groups jointly launched a challenge to assess the reproducibility of a well-established inversion-recovery T₁ mapping technique, using acquisition details from a seminal T₁ mapping paper on a standardized phantom and in human brains.

Methods: The challenge used the acquisition protocol from Barral et al. (2010). Researchers collected T₁ mapping data on the ISMRM/NIST phantom and/or in human brains. Data submission, pipeline development, and analysis were conducted using open-source platforms. Intersubmission and intrasubmission comparisons were performed.

Results: Eighteen submissions (39 phantom and 56 human datasets) on scanners by three MRI vendors were collected at 3 T (except one, at 0.35 T). The mean coefficient of variation was 6.1% for intersubmission phantom measurements, and 2.9% for intrasubmission measurements. For humans, the intersubmission/intrasubmission coefficient of variation was 5.9/3.2% in the genu and 16/6.9% in the cortex. An interactive dashboard for data visualization was also developed: https://rrsg2020.dashboards.neurolibre.org.

Conclusion: The T₁ intersubmission variability was twice as high as the intrasubmission variability in both phantoms and human brains, indicating that the acquisition details in the original paper were insufficient to reproduce a quantitative MRI protocol. This study reports the inherent uncertainty in T₁ measures across independent research groups, bringing us one step closer to a practical clinical baseline of T₁ variations in vivo.

Keywords: T1 mapping; inversion recovery; open data; quantitative MRI; reproducibility.