Challenges and caveats of a multi-center retrospective radiomics study: an example of early treatment response assessment for NSCLC patients using FDG-PET/CT radiomics

PLoS One. 2019 Jun 3;14(6):e0217536. doi: 10.1371/journal.pone.0217536. eCollection 2019.

Abstract

Background: Prognostic models based on individual patient characteristics can improve treatment decisions and outcome in the future. In many (radiomic) studies, small size and heterogeneity of datasets is a challenge that often limits performance and potential clinical applicability of these models. The current study is example of a retrospective multi-centric study with challenges and caveats. To highlight common issues and emphasize potential pitfalls, we aimed for an extensive analysis of these multi-center pre-treatment datasets, with an additional 18F-fluorodeoxyglucose (FDG) positron emission tomography/computed tomography (PET/CT) scan acquired during treatment.

Methods: The dataset consisted of 138 stage II-IV non-small cell lung cancer (NSCLC) patients from four different cohorts acquired from three different institutes. The differences between the cohorts were compared in terms of clinical characteristics and using the so-called 'cohort differences model' approach. Moreover, the potential prognostic performances for overall survival of radiomic features extracted from CT or FDG-PET, or relative or absolute differences between the scans at the two time points, were assessed using the LASSO regression method. Furthermore, the performances of five different classifiers were evaluated for all image sets.

Results: The individual cohorts substantially differed in terms of patient characteristics. Moreover, the cohort differences model indicated statistically significant differences between the cohorts. Neither LASSO nor any of the tested classifiers resulted in a clinical relevant prognostic model that could be validated on the available datasets.

Conclusion: The results imply that the study might have been influenced by a limited sample size, heterogeneous patient characteristics, and inconsistent imaging parameters. No prognostic performance of FDG-PET or CT based radiomics models can be reported. This study highlights the necessity of extensive evaluations of cohorts and of validation datasets, especially in retrospective multi-centric datasets.

Publication types

  • Clinical Trial
  • Comparative Study
  • Multicenter Study
  • Research Support, Non-U.S. Gov't

MeSH terms

  • Aged
  • Carcinoma, Non-Small-Cell Lung* / diagnostic imaging
  • Carcinoma, Non-Small-Cell Lung* / mortality
  • Carcinoma, Non-Small-Cell Lung* / therapy
  • Databases, Factual*
  • Disease-Free Survival
  • Female
  • Fluorodeoxyglucose F18 / administration & dosage*
  • Humans
  • Lung Neoplasms* / diagnostic imaging
  • Lung Neoplasms* / mortality
  • Lung Neoplasms* / therapy
  • Male
  • Middle Aged
  • Models, Biological*
  • Positron Emission Tomography Computed Tomography*
  • Retrospective Studies
  • Survival Rate

Substances

  • Fluorodeoxyglucose F18

Grants and funding

Authors acknowledge financial support from ERC advanced grant (ERC-ADG-2015, n° 694812 - Hypoximmuno) and the QuIC-ConCePT project, which is partly funded by EFPI A companies and the Innovative Medicine Initiative Joint Undertaking (IMI JU) under Grant Agreement No. 115151. This research is also supported by the Dutch technology Foundation STW (n° P14-19 Radiomics STRaTegy), which is the applied science division of NWO, and the Technology Programme of the Ministry of Economic Affairs. Authors also acknowledge financial support from SME Phase 2 (RAIL - n°673780), EUROSTARS (DART, DECIDE,), the European Program H2020-2015-17 (ImmunoSABR - n° 733008 and PREDICT - ITN - n° 766276), Interreg V-A Euregio Meuse-Rhine (“Euradiomics”) and Kankeronderzoekfonds Limburg from the Health Foundation Limburg. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.