Quality assuring the quality assurance tool: applying safety-critical concepts to test framework development

PeerJ Comput Sci. 2022 Oct 28:8:e1131. doi: 10.7717/peerj-cs.1131. eCollection 2022.


The quality of embedded systems is demonstrated by the performed tests. The quality of such tests is often dependent on the quality of one or more testing tools, especially in automated testing. Test automation is also central to the success of agile development. It is thus critical to ensure the quality of testing tools. This work explores how industries with agile processes can learn from safety-critical system development with regards to the quality assurance of the test framework development. Safety-critical systems typically need adherence to safety standards that often suggests substantial upfront documentation, plans and a long-term perspective on several development aspects. In contrast, agile approaches focus on quick adaptation, evolving software and incremental deliveries. This article identifies several approaches of quality assurance of software development tools in functional safety development and agile development. The extracted approaches are further analyzed and processed into candidate solutions, i.e., principles and practices for the test framework quality assurance applicable in an industrial context. An industrial focus group with experienced practitioners further validated the candidate solutions through moderated group discussions. The two main contributions from this study are: (i) 48 approaches and 25 derived candidate solutions for test framework quality assurance in four categories (development, analysis, run-time measures, and validation and verification) with related insights, e.g., a test framework should be perceived as a tool-chain and not a single tool, (ii) the perceived value of the candidate solutions in industry as collected from the focus group.

Keywords: Agile processes; Case study; Hybrid processes; Quality assurance; Safety-critical development; Test automation.

Grants and funding

This research was funded by Westermo Network Technologies AB, the Knowledge Foundation grant 20150277 (ITS ESS-H), and the European Union’s Horizon 2020 research and innovation program under grant agreement nos. 871319 & 957212. There was no additional external funding received for this study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.