PhD Thesis within TRUCONF

Richard Schumi has written his PhD thesis about testing and performance evaluation techniques that were developed within TRUCONF:

Richard Schumi: “Predicting and Testing System Response-Times with Statistical Model Checking and Property-Based Testing”, Graz University of Technology, Institute for Software Technology, 2018. (PDF)

The work was supervised by Bernhard K. Aichernig (Graz University of Technology) and reviewed by John Hughes (Chalmers University of Technology), and it was successfully defended on Nov 9th, 2018 with distinction.

Abstract:

In recent years, software systems have become increasingly omnipresent. Especially since the Internet of Things is spreading into the everyday life of millions of people. However, these systems often fail to satisfy the requirements of the users, both in terms of functionality and performance. Hence, in order to increase user satisfaction, it is essential to perform a rigorous quality-assurance process, most importantly in the form of software testing.

A testing technique that became popular in recent years is property-based testing. This technique relies on randomly generated test data in order to check if functions- or systems-under-test work as expected. Moreover, it can perform a test-case generation that is based on a behavioural model of a system. The definition of such a model normally requires a high manual effort. To deal with this issue, we present a testing technique that works with business-rule models that are existing system artefacts. Hence, there is no need for a manual model definition. We apply this method in order to find bugs, to increase the confidence in the functionality of the system, and also to produce log data, which we apply for performance testing.

Performance testing comprises a group of techniques that aim to evaluate performance requirements, like responsiveness or scalability of a system. It usually involves a high number of tests that are directly executed on a system, which becomes especially cumbersome, when different usage scenarios should be considered. Therefore, we propose a model-based method that works with a fast simulation to predict the expected response times for users.

First, we run property-based testing concurrently to obtain log data from simultaneous system interactions. Based on this data, we learn a stochastic model that we apply for statistical model checking in order to receive predictions with a certain confidence. Moreover, we propose an efficient evaluation technique for such predictions. By performing hypothesis testing on the system, we can check the accuracy of our model with fewer samples than needed for the model simulation.

Our method is realised in a property-based testing tool that we have enhanced with algorithms from statistical model checking. This tool allows both, simulating stochastic models and testing the simulation results directly on a system. We demonstrate the feasibility with an industrial case study of a web-service application and by performing a comparison of two protocol implementations from the Internet of Things.

Keywords:

Property-Based Testing, Statistical Model Checking, Model-Based Testing, Performance, Response Time, Latency, Web-Service Application, Business-Rule Models, Internet of Things, MQTT, FsCheck.