Write a Blog >>
ICST 2022
Mon 4 - Fri 8 April 2022
Fri 8 Apr 2022 15:10 - 15:30 at Marlyn Meltzer - NEXTA III Chair(s): Sahar Tahvili

As stability testing execution logs can be very long, software engineers need help in locating anomalous events to investigate when failures occur. We develop and evaluate two models for scoring individual log-events for anomalousness, namely an N-Gram model and a Deep Learning model with LSTM cells. Both are trained on normal log sequences only. We evaluate the models with long log sequences produced by stability testing in our company case and contrast that with very short log sequences from HDFS public dataset. We evaluate the prediction accuracy of the next event and computational efficiency. The N-Gram model is more accurate in stability testing logs (0.848 vs 0.831), whereas almost identical accuracy is seen in HDFS logs (0.849 vs 0.847). The N-Gram model has superior computational efficiency compared to the Deep model (4 to 13 seconds vs 16 minutes to nearly 4 hours), making it the preferred choice for our case company. Scoring individual log events for anomalousness seems like a good aid for root cause analysis of failing test cases, and our case company plans to add it to its online services.
Despite the recent surge in using deep learning in software system anomaly detection, we found no benefits in doing so. Future work should consider whether this holds with different model configurations, other datasets, and with other deep-learning approaches that promise better accuracy and computational efficiency than LSTM based models.

Fri 8 Apr

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

15:10 - 15:50
NEXTA IIINEXTA at Marlyn Meltzer
Chair(s): Sahar Tahvili Ericsson AB
Pinpointing Anomaly Events in Logs from Stability Testing - N-Grams vs. Deep-Learning
Mika Mäntylä University of Oulu, Martin Varela Profilence, Shayan Hashemi University of Oulu
Identifying Randomness related Flaky Tests through Divergence and Execution Tracing
Azeem Ahmad Linköping University, Erik Norrestam Held , Ola Leifler , Kristian Sandahl Linköping University