External validation of SAPS 3 and MPM0-III scores in 48,816 patients from 72 Brazilian ICUs

Giulliana Martines Moralez; Ligia Sarmet Cunha Farah Rabello; Thiago Costa Lisboa; Mariza da Fonte Andrade Lima; Rodrigo Marques Hatum; Fernando Vinicius Cesar De Marco; Alessandra Alves; Jorge Eduardo da Silva Soares Pinto; Hélia Beatriz Nunes de Araújo; Grazielle Viana Ramos; Aline Reis Silva; Guilherme Côrtes Fernandes; Guilherme Brenande Alves Faria; Ciro Leite Mendes; Roberto Álvaro Ramos Filho; Valdênia Pereira de Souza; Pedro Emmanuel Alvarenga Americano do Brasil; Fernando Augusto Bozza; Jorge Ibrain Figueira Salluh; Marcio Soares; ORCHESTRA Study Investigators

doi:10.1186/s13613-017-0276-3

External validation of SAPS 3 and MPM₀-III scores in 48,816 patients from 72 Brazilian ICUs

Ann Intensive Care. 2017 Dec;7(1):53. doi: 10.1186/s13613-017-0276-3. Epub 2017 May 18.

Authors

Giulliana Martines Moralez¹, Ligia Sarmet Cunha Farah Rabello^{2

3}, Thiago Costa Lisboa⁴, Mariza da Fonte Andrade Lima⁵, Rodrigo Marques Hatum⁶, Fernando Vinicius Cesar De Marco⁷, Alessandra Alves⁸, Jorge Eduardo da Silva Soares Pinto⁹, Hélia Beatriz Nunes de Araújo¹⁰, Grazielle Viana Ramos¹¹, Aline Reis Silva¹¹, Guilherme Côrtes Fernandes¹², Guilherme Brenande Alves Faria¹³, Ciro Leite Mendes¹⁴, Roberto Álvaro Ramos Filho¹⁵, Valdênia Pereira de Souza¹⁶, Pedro Emmanuel Alvarenga Americano do Brasil^{11

17}, Fernando Augusto Bozza^{11

17}, Jorge Ibrain Figueira Salluh^{2

18}, Marcio Soares^{19

20

21}; ORCHESTRA Study Investigators

Affiliations

¹ Graduate Program in Translational Medicine, D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil.
² PPG Internal Medicine, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil.
³ ICU, Hospital Copa D'Or, Rio de Janeiro, Brazil.
⁴ Complexo Hospitalar, Santa Casa de Misericórdia de Porto Alegre, Porto Alegre, Brazil.
⁵ ICU, Hospital Esperança Recife, Recife, Brazil.
⁶ ICU, Hospital Total Cor, Rio de Janeiro, Brazil.
⁷ ICU, Hospital viValle, São José dos Campos, Brazil.
⁸ ICU, Hospital Rios D'Or, Rio de Janeiro, Brazil.
⁹ ICU, Hospital Norte D'Or, Rio de Janeiro, Brazil.
¹⁰ Hospital do Coração do Brasil, Brasília, Brazil.
¹¹ Department of Critical Care, D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil.
¹² ICU, Santa Casa de Misericórdia de Juiz de Fora, Juiz de Fora, Brazil.
¹³ ICU, Hospital Oeste D'Or, Rio de Janeiro, Brazil.
¹⁴ ICU, Hospital Universitário Lauro Wanderley, João Pessoa, Brazil.
¹⁵ ICU, Hospital São Luiz - Unidade Jabaquara, São Paulo, Brazil.
¹⁶ Complexo Hospitalar de Niterói, Niterói, Brazil.
¹⁷ Instituto Nacional de Infectologia Evandro Chagas, Fundação Oswaldo Cruz-Fiocruz, Rio de Janeiro, Brazil.
¹⁸ Department of Critical Care and Graduate Program in Translational Medicine, D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil.
¹⁹ PPG Internal Medicine, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil. marciosoaresms@gmail.com.
²⁰ Department of Critical Care and Graduate Program in Translational Medicine, D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil. marciosoaresms@gmail.com.
²¹ Department of Critical Care, D'Or Institute for Research and Education, Rua Diniz Cordeiro, 30. Botafogo, Rio de Janeiro, 22281-100, Brazil. marciosoaresms@gmail.com.

Abstract

Background: The performance of severity-of-illness scores varies in different scenarios and must be validated prior of being used in a specific settings and geographic regions. Moreover, models' calibration may deteriorate overtime and performance of such instruments should be reassessed regularly. Therefore, we aimed at to validate the SAPS 3 in a large contemporary cohort of patients admitted to Brazilian ICUs. In addition, we also compared the performance of the SAPS 3 with the MPM₀-III.

Methods: This is a retrospective cohort study in which 48,816 (medical admissions = 67.9%) adult patients are admitted to 72 Brazilian ICUs during 2013. We evaluated models' discrimination using the area under the receiver operating characteristic curve (AUROC). We applied the calibration belt to evaluate the agreement between observed and expected mortality rates (calibration).

Results: Mean SAPS 3 score was 44.3 ± 15.4 points. ICU and hospital mortality rates were 11.0 and 16.5%. We estimated predicted mortality using both standard (SE) and Central and South American (CSA) customized equations. Predicted mortality rates were 16.4 ± 19.3% (SAPS 3-SE), 21.7 ± 23.2% (SAPS 3-CSA) and 14.3 ± 14.0% (MPM₀-III). Standardized mortality ratios (SMR) obtained for each model were: 1.00 (95% CI, 0.98-0.102) for the SAPS 3-SE, 0.75 (0.74-0.77) for the SAPS 3-CSA and 1.15 (1.13-1.18) for the MPM₀-III. Discrimination was better for SAPS 3 models (AUROC = 0.85) than for MPM₀-III (AUROC = 0.80) (p < 0.001). We applied the calibration belt to evaluate the agreement between observed and expected mortality rates (calibration): the SAPS 3-CSA overestimated mortality throughout all risk classes while the MPM₀-III underestimated it uniformly. The SAPS 3-SE did not show relevant deviations from ideal calibration.

Conclusions: In a large contemporary database, the SAPS 3-SE was accurate in predicting outcomes, supporting its use for performance evaluation and benchmarking in Brazilian ICUs.

Keywords: Intensive care units; Outcomes; Severity-of-illness scores; Standardized mortality rate; Validation.