Background: The negative psychosocial impacts of cancer diagnoses and treatments are well documented. Virtual care has become an essential mode of care delivery during the COVID-19 pandemic, and online support groups (OSGs) have been shown to improve accessibility to psychosocial and supportive care. de Souza Institute offers CancerChatCanada, a therapist-led OSG service where sessions are monitored by an artificial intelligence-based co-facilitator (AICF). The AICF is equipped with a recommender system that uses natural language processing to tailor online resources to patients according to their psychosocial needs.
Objective: We aimed to outline the development protocol and evaluate the AICF on its precision and recall in recommending resources to cancer OSG members.
Methods: Human input informed the design and evaluation of the AICF on its ability to (1) appropriately identify keywords indicating a psychosocial concern and (2) recommend the most appropriate online resource to the OSG member expressing each concern. Three rounds of human evaluation and algorithm improvement were performed iteratively.
Results: We evaluated 7190 outputs and achieved a precision of 0.797, a recall of 0.981, and an F1 score of 0.880 by the third round of evaluation. Resources were recommended to 48 patients, and 25 (52%) accessed at least one resource. Of those who accessed the resources, 19 (75%) found them useful.
Conclusions: The preliminary findings suggest that the AICF can help provide tailored support for cancer OSG members with high precision, recall, and satisfaction. The AICF has undergone rigorous human evaluation, and the results provide much-needed evidence, while outlining potential strengths and weaknesses for future applications in supportive care.
Keywords: artificial intelligence; natural language processing; online support groups; recommender system; supportive care in cancer.
©Yvonne W Leung, Bomi Park, Rachel Heo, Achini Adikari, Suja Chackochan, Jiahui Wong, Elyse Alie, Mathew Gancarz, Martyna Kacala, Graeme Hirst, Daswin de Silva, Leon French, Jacqueline Bender, Faye Mishna, David Gratzer, Damminda Alahakoon, Mary Jane Esplen. Originally published in JMIR Cancer (https://cancer.jmir.org), 29.07.2022.