Vigi4Med Scraper: A Framework for Web Forum Structured Data Extraction and Semantic Representation

Bissan Audeh; Michel Beigbeder; Antoine Zimmermann; Philippe Jaillon; Cédric Bousquet

doi:10.1371/journal.pone.0169658

Vigi4Med Scraper: A Framework for Web Forum Structured Data Extraction and Semantic Representation

PLoS One. 2017 Jan 25;12(1):e0169658. doi: 10.1371/journal.pone.0169658. eCollection 2017.

Authors

Bissan Audeh¹, Michel Beigbeder¹, Antoine Zimmermann¹, Philippe Jaillon², Cédric Bousquet³

Affiliations

¹ University of Lyon, MINES Saint-Étienne, CNRS, Hubert Curien Laboratory, UMR 5516, Saint-Étienne, France.
² Ecole Nationale Supérieure des Mines de Saint-Étienne, Saint-Étienne, France.
³ INSERM, U1142, LIMICS, Paris, France.

Abstract

The extraction of information from social media is an essential yet complicated step for data analysis in multiple domains. In this paper, we present Vigi4Med Scraper, a generic open source framework for extracting structured data from web forums. Our framework is highly configurable; using a configuration file, the user can freely choose the data to extract from any web forum. The extracted data are anonymized and represented in a semantic structure using Resource Description Framework (RDF) graphs. This representation enables efficient manipulation by data analysis algorithms and allows the collected data to be directly linked to any existing semantic resource. To avoid server overload, an integrated proxy with caching functionality imposes a minimal delay between sequential requests. Vigi4Med Scraper represents the first step of Vigi4Med, a project to detect adverse drug reactions (ADRs) from social networks founded by the French drug safety agency Agence Nationale de Sécurité du Médicament (ANSM). Vigi4Med Scraper has successfully extracted greater than 200 gigabytes of data from the web forums of over 20 different websites.

MeSH terms

Algorithms*
Data Collection / methods*
Humans
Internet*
Social Media

Grants and funding

This work was supported by grant number AAP-2013-052 from the ANSM, the French agency for drug safety (Agence nationale de sécurité du médicament et des produits de santé). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.