Capturing customer workloads of database systems to replay them during internal testing can be beneficial for software quality assurance. However, we experienced at SAP that such replays can produce a large amount of false positive alerts that make the results unreliable or time consuming to analyze. Therefore, we designed a machine learning based approach that attributes root causes to the alerts. This provides several benefits for quality assurance and allows for example to classify whether an alert is true positive or false positive. Our approach considerably reduces manual effort and allows SAP to improve the overall quality assurance for the database system SAP HANA. We discuss the problem, the design and result of our approach, and we present practical limitations that may require further research.