As Code Testing: Characterizing Test Quality in Open Source Ansible Development
Infrastructure as code (IaC) scripts, such as Ansible scripts, are used to provision computing infrastructure at scale. Existence of bugs in IaC test scripts, such as, configuration and security bugs, can be consequential for the provisioned computing infrastructure. A characterization study of bugs in IaC test scripts is the first step to understand the quality concerns that arise during testing of IaC scripts, and also provide recommendations for practitioners on quality assurance. We conduct an empirical study with 4,831 Ansible test scripts mined from 104 open source software (OSS) repositories where we quantify bug frequency, and categorize bugs in test scripts. We further categorize testing patterns, i.e., recurring coding patterns in test scripts, which also correlate with appearance of bugs. From our empirical study, we observe 1.8% of 4,831 Ansible test scripts to include a bug, and 45.2% of the 104 repositories to contain at least one test script that includes bugs. We identify 7 categories of bugs, which includes security bugs, and performance bugs that are related with metadata extraction. We also identify 3 testing patterns that correlate with appearance of bugs: ‘assertion roulette’, ‘local only testing’, and ‘remote mystery guest’. We conclude our paper by discussing the implications of our findings for practitioners.
Presentation (TEASER-AS-CODE-TESTING.mp4) | 2.52MiB |
Presentation Slides (ICST2022-V2.pptx) | 13.97MiB |
Wed 6 AprDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:45 - 18:00 | ICST Empirical StudyJournal-First Papers / Research Papers at Margaret Hamilton Chair(s): Mohammad Reza Mousavi King's College London | ||
16:45 15mTalk | An experimental and practical study on the equivalent mutant connection: An evolutionary approach Journal-First Papers Link to publication DOI | ||
17:00 15mTalk | A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests Research Papers Sarra Habchi University of Luxembourg, Luxembourg, Guillaume Haben University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg Pre-print | ||
17:15 15mTalk | As Code Testing: Characterizing Test Quality in Open Source Ansible Development Research Papers Pre-print File Attached | ||
17:30 15mTalk | A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It Research Papers Pre-print | ||
17:45 15mLive Q&A | Discussion and Q&A Research Papers |