Write a Blog >>
ICST 2022
Mon 4 - Fri 8 April 2022

Gordon Fraser will present the keynote on

“On the Challenges of Building and Evaluating Test Generators”

IWCT 2022 is to be held in conjunction with ICST 2022 (going virtual), focusing on combinatorial testing. The workshop welcomes academic research submissions, as well as industrial experience reports.

Combinatorial Testing (CT), or Combinatorial Interaction Testing (CIT), is a widely applicable generic methodology and technology for software verification and validation, considered a testing best practice. In a combinatorial test plan, all interactions between parameters up to a certain level are covered. For example, in pairwise testing, for every pair of parameters, every pair of values will appear at least once. Studies show that CT is more efficient and effective than random testing.

CT has gained significant interest in recent years, both in research and in practice. However, many issues still remain unresolved, and much research is still needed in the field. For example, while pairwise testing is a well recognized and popular test planning method, investigations of actual failures in a number of software and systems convincingly show that pairwise testing is usually not sufficient, so high strength CT (i.e., t-way for t > 2) may be needed.

In addition, the combinatorial test suites need to exclude invalid combinations of test values that cannot be executed, which limits the degrees of freedom the algorithms have, thus complicating the problem. Moreover, modeling languages and tools for easily capturing the input test space are also required for real-life applicability of CT. Other obstacles for wide acceptance of CT in industry are the gap between the generated test plans and executable tests, and the difficulty in determining expected results for the generated tests. Finally, empirical studies on CT, as well as thorough comparison with other methods are also required.

In this workshop, we plan to bring together researchers actively working on combinatorial testing, and create a productive and creative environment for sharing and collaboration. Since there is no other venue dedicated to CT, yet there are many researchers working in the field, we expect, like in previous years, to see high responsiveness to take part in the workshop. Researchers attending the workshop will have an opportunity to publish their work in a dedicated venue, create new collaborations and take active part in the growing community of researchers working in the field.

The workshop will also be a meeting place between academia and industry, thus uniting academic excellence and industrial experience and needs. This will allow participants from academia to learn about the industrial experience in practical application of CT to real-life testing problems, and together with the colleagues from industry identify the difficulties that are obstacle to wider application of CT, and should be addressed in future research. Industrial participants will have an opportunity to meet the leading scientists in the field, and hear about the latest advances and innovations.

Combinatorial test competition

IWCT will host a competition among tools for combinatorial testing - We are still working on the details, some preliminary information is here: https://fmselab.github.io/ct-competition/ If you are interested in helping us in defining the rules, please let us know.

You're viewing the program in a time zone which is different from your device's time zone change time zone

Mon 4 Apr

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

09:00 - 09:10
IWCT OpeningIWCT at Grace Hopper
09:00
10m
Day opening
Opening
IWCT

09:10 - 10:10
IWCT KeynoteIWCT at Grace Hopper
09:10
60m
Keynote
On the Challenges of Building and Evaluating Test Generators
IWCT
Gordon Fraser University of Passau
10:30 - 12:10
IWCT Research Papers IIWCT at Grace Hopper
10:30
25m
Talk
Applying Combinatorial Testing to High-Speed Railway Automatic Train Protection System
IWCT
Rui Huang Southwest Jiaotong University, Chang Rao Chongqing Jiaotong University, Jeff Yu Lei University of Texas at Arlington, Jin Guo Southwest Jiaotong University, Yadong Zhang Southwest Jiaotong University
10:55
25m
Talk
Using Unified Combinatorial Interaction Testing for MC/DC Coverage
IWCT
Giray Coskun Sabanci University, Cankut Coskun Sabanci University, Hanefi Mercan Sabanci University, Cemal Yilmaz Sabancı University
11:20
25m
Talk
CA^2: Practical Archival and Compression of Covering Arrays
IWCT
Manuel Leithner SBA Research, Dimitris Simos SBA Research
11:45
25m
Talk
Experience of Combinatorial Testing toward Fault Detection, Isolation and Recovery Functionality
IWCT
Naoko Okubo Japan Aerospace Exploration Agency, Japan, Shoma Takatsuki Japan Aerospace Exploration Agency, Yasushi Ueda Japan Aerospace Exploration Agency, Japan
13:00 - 14:40
IWCT Tool CompetitionIWCT at Grace Hopper
13:00
25m
Talk
A C++ implementation of the IPO algorithm
IWCT
Ken'Ya Takemura Osaka University Graduate School of Information Science and Technology
13:25
25m
Talk
Parallel Test Generation for Combinatorial Models Based on Multivalued Decision Diagrams
IWCT
Andrea Bombarda University of Bergamo, Angelo Gargantini University of Bergamo
13:50
25m
Talk
A Constrained Covering Array Generator using Adaptive Penalty based Parallel Tabu Search
IWCT
yan wang Nanjing University, Huayao Wu Nanjing University, Changhai Nie Nanjing University, Xintao Niu Nanjing University, Jiaxi Xu School of Information Engineering, Nanjing Xiaozhuang University
14:15
25m
Talk
Developing multithreaded techniques and improved constraint handling for the tool CAgen
IWCT
Michael Wagner SBA Research, Manuel Leithner SBA Research, Dimitris Simos SBA Research, Rick Kuhn National Institute of Standards and Technology, Raghu Kacker National Institute of Standards and Technology
15:00 - 16:15
IWCT Research Papers IIIWCT at Grace Hopper
15:00
25m
Talk
A Combinatorial Approach to Fairness Testing of Machine Learning Models
IWCT
Ankita Ramjibhai Patel The University of Texas at Arlington, Jaganmohan Chandrasekaran Virginia Tech, Jeff Yu Lei University of Texas at Arlington, Raghu Kacker National Institute of Standards and Technology, Rick Kuhn National Institute of Standards and Technology
15:25
25m
Talk
Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing
IWCT
Tyler Cody Virginia Tech, Erin Lanus The MITRE Corporation, Daniel D. Doyle Virginia Tech, Laura Freeman Virginia Tech
15:50
25m
Talk
Combination Frequency Differencing for Identifying Design Weaknesses in Physical Unclonable Functions
IWCT
Rick Kuhn National Institute of Standards and Technology, M S Raunak National Institute of Standards and Technology, Charles Prado National Institute of Metrology, Quality and Technology, Vinay Patil University of Massachusetts Amherst, Raghu Kacker National Institute of Standards and Technology
16:15 - 16:40
IWCT ClosingIWCT at Grace Hopper
16:15
15m
Awards
Results of the CT competition
IWCT
Manuel Leithner SBA Research, Andrea Bombarda University of Bergamo, Michael Wagner SBA Research
16:30
10m
Day closing
Closing
IWCT

Accepted Papers

Title
A C++ implementation of the IPO algorithm
IWCT
A Combinatorial Approach to Fairness Testing of Machine Learning Models
IWCT
A Constrained Covering Array Generator using Adaptive Penalty based Parallel Tabu Search
IWCT
Applying Combinatorial Testing to High-Speed Railway Automatic Train Protection System
IWCT
CA^2: Practical Archival and Compression of Covering Arrays
IWCT
Combination Frequency Differencing for Identifying Design Weaknesses in Physical Unclonable Functions
IWCT
Developing multithreaded techniques and improved constraint handling for the tool CAgen
IWCT
Experience of Combinatorial Testing toward Fault Detection, Isolation and Recovery Functionality
IWCT
Parallel Test Generation for Combinatorial Models Based on Multivalued Decision Diagrams
IWCT
Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing
IWCT
Using Unified Combinatorial Interaction Testing for MC/DC Coverage
IWCT

Call for Papers

We invite submissions of high-quality papers presenting original work on both theoretical and experimental aspects of combinatorial testing.

Full and Short Papers

Papers should not exceed 8 pages for full papers or 4 pages for short experience and position papers, excluding references - but it is not a strict limit, if you need more space contact the chairs. Each submitted paper must conform to the IEEE two-column publication format. Papers will be reviewed by at least three members from the program committee. Accepted papers will be published in the IEEE Digital Library.

CT competition papers

Participants to the CT competition (https://fmselab.github.io/ct-competition/) are invited to submit a paper in which they discuss their tool and results (against the benchamark provided in advance wrt the competition itself).

Journal First

The aim of journal-first papers in category is to further enrich the program of IWCT, as well as to provide an overall more flexible path to publication and dissemination of original research in model-based testing. The published journal paper must adhere to the following three criteria:

  • It should be clearly within the scope of the workshop.
  • It should be recent: it should have been accepted and made publicly available in a journal (online or in print) by 1 January 2020 or more recently.
  • It has not been presented at, and is not under consideration for, journal-first tracks of other conferences or workshops.

The 2-page submission should provide a concise summary of the published journal paper.

Journal-first submissions must be marked as such in the submission’s title, and must explicitly include full bibliographic details (including a DOI) of the journal publication they are based on. Submissions will be judged on the basis of the above criteria, but also considering how well they would complement the workshop’s technical program.

TOPICS

Topics of interest for full and short papers include, but are not limited to:

Combinatorial testing workflow

  • Modeling the input space for CT – the input to CT algorithms is a set of parameters, respective values, and constraints on value combinations. This input should capture correctly the points of variability in testing the system. While this input is clearly crucial for the effectiveness of CT, it is a difficult problem.
  • Efficient algorithms to generate test suites with small size for t-way testing for t > 2, involving support of constraints on combinations that are possible.
  • Determination of expected system behavior for each test case – while the test cases are automatically generated by the CT algorithm, determining the expected system behavior is currently usually a manual task.
  • Executing CT test suites – the result of CT algorithms is a list of tests, where a test is represented by a value for each parameter. There is still significant effort in transforming this representation into actual tests that a tester or testing tool can execute.
  • Combinatorial testing based fault localization.
  • Implementation of CT in existing testing infrastructures.
  • Handling changes in test requirements – current CT methods focus on one-time generation of a test plan from a given set of requirements. However, test requirements are almost never static, and tend to change between different releases and versions. On the other hand, generating a new set of tests for each release may not be practical either.

Real-world experience in deployment of combinatorial testing

  • Empirical studies and feedback from practical applications of CT.
  • Evaluation and return of investment metrics to assess the degree of usefulness of CT.
  • Methodology used for test space modeling and determination of interaction coverage requirements.
  • Discussion of challenges and open problems in the application of CT in industrial settings.
  • Combinatorial testing for AI-based systems.

Applicability of combinatorial testing

  • Comparison and combination of CT with other dynamic verification methods.
  • Investigation of historical records of failures to determine the kind of CT which may have detected underlying faults.
  • Combinatorial testing for concurrent and real-time systems.
  • CT for testing cloud computing systems and use of combinatorial methods in cloud architecture.
  • Application of CT in other domains, e.g., information security, study of gene regulation and other biotechnology applications and mechanical engineering.
  • Combinatorial testing of feature models for software product lines.

Combinatorial and complementing methods

  • Combinatorial analysis of existing test suites – analyze a test suite not generated by a CT algorithm in light of a combinatorial test space.
  • Test plan reduction and completeness – both under stable, and under changing test requirements.
  • CT and coverage metrics – combining the two, and studying the relation between them.

Questions? Use the IWCT contact form.