BigDataScript: a scripting language for data pipelines

Abstract

Motivation: The analysis of large biological datasets often requires complex processing pipelines that run for a long time on large computational infrastructures. We designed and implemented a simple script-like programming language with a clean and minimalist syntax to develop and manage pipeline execution and provide robustness to various types of software and hardware failures as well as portability.

Results: We introduce the BigDataScript (BDS) programming language for data processing pipelines, which improves abstraction from hardware resources and assists with robustness. Hardware abstraction allows BDS pipelines to run without modification on a wide range of computer architectures, from a small laptop to multi-core servers, server farms, clusters and clouds. BDS achieves robustness by incorporating the concepts of absolute serialization and lazy processing, thus allowing pipelines to recover from errors. By abstracting pipeline concepts at programming language level, BDS simplifies implementation, execution and management of complex bioinformatics pipelines, resulting in reduced development and debugging cycles as well as cleaner code.

Availability and implementation: BigDataScript is available under open-source license at http://pcingola.github.io/BigDataScript.

Listing 1.

pipeline.bds program. A simple pipeline example featuring a task invoking a fictitious command ‘myProcess’ defined to require 2 CPUs and a maximum of 6 h of execution time (Line 5)

Listing 2.

pipeline_2.bds program. A two-step pipeline with task dependencies. The first step (line 9) requires to run ‘myProcess’ command on a hundred input files, which can be executed in parallel. The second step (line 19) processes the output of those hundred files and creates a single output file (using fictitious ‘myProcessAll’ command). It should be noted that we never explicitly state which hardware we are using: (i) if the pipeline is run on a dual-core computer, as each process requires 2 CPUs, one ‘myProcess’ instance will be executed at the time until the 100 tasks are completed; (ii) if it is run on a 64-core server, then 32 ‘myProcess’ instances will be executed in parallel; (iii) if it is run on a cluster, then 100 ‘myProcess’ instances will be scheduled and the cluster resource management system will decide how to execute them; and (iv) if it is run on a single-core computer, execution will fail owing to lack of resources. Thus, the pipeline runs independent of the underlying architecture. The task defined in line 18 depends on all the outputs from tasks in line 8 (‘mainOut <− outs’)

Fig. 1.

BDS report showing pipeline’s task execution timeline

Fig. 2.

Execution example. (A) Script ‘pipeline.bds’. (B) The script is executed from a terminal. The GO executable invokes main BDS, written in JAVA, performs lexing, parsing, compilation to AST and runs AST. (C) When the task statement is run, appropriate checks are performed. (D) A shell script ‘task1.sh’ is created, and a bds-exec process is fired. (E) bds-exec reports PID, executed the script ‘task1.sh’ while capturing stdout and stderr as well as monitoring timeouts and OS signals. When a process finishes execution, the exit status is logged

Fig. 3.

Whole-genome sequencing analysis pipeline’s flow chart, showing how computations are split across many nodes