As a concluding event for the project, we held a workshop at AVL on the 31th January 2018. During this workshop, we presented the methods developed within the project and the obtained evaluation results to an audience of AVL staff members. We arranged two sessions, one for each use case that was tackled within TRUCONF.
The major outcome of the first use case that was concerned with testing measurement devices, was a textual domain-specific modelling language called MDML. This language is easy to learn and to use so that non-experts can create models for model-based test-case generation [Burghard et al.].
The second use case targeted performance testing of a web-service-application for test bed management, called Test Factory Management Suite (TFMS). For this use case, we presented a testing technique that combines two existing methods property-based testing and statistical model checking in order to predict and check if the system requests satisfies certain response time limits [Schumi et al.].
We got a lot of positive feedback for our work and our results were praised by AVL. Moreover, the success of the project laid the foundation for future collaboration and projects.
Project Team at the Final Workshop
Silvio Marcovic contributed with his Master’s Thesis to the test-case generation methods and tools of TRUCONF.
Silvio Marcovic: “Integrating an External Test-Case Generator into a Property-Based Testing Tool for Testing Web-Services“. Graz University of Technology, Institute for Software Technology, 2017. (PDF)
The work was supervised by Bernhard K. Aichernig and co-supervised by Richard Schumi. It was defended on the 9th November 2017.
A flyer that summarises the results of TFUCONF and the achievements within the project has been created. (PDF)
A publication has been accepted at the MODELSWARD 2018.
Gerald Stieglbauer, Christian Burghard, Stefan Sobernig and Robert Korošec: “A Daily Dose of DSL – MDE Micro Injections in Practice” In Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development (MODELSWARD 2018), Funchal, Portugal, 22-24 January 2018, pages 642-651, SCITEPRESS, 2018. (PDF)
The paper will be presented at the 6th International Conference on Model-Driven Engineering and Software Development (MODELSWARD 2018) in Funchal, Portugal (22-24 January 2018).
Another publication has been accepted at the International Journal on Software and Systems Modeling (SoSyM).
Bernhard K. Aichernig and Richard Schumi: “Property-Based Testing of Web Services by Deriving Properties from Business-Rule Models”. International Journal on Software and Systems Modeling (SoSyM), 2017, Open Access. (PDF) (doi:10.1007/s10270-017-0647-0)
Property-based testing is well suited for web-service applications, which was already shown in various case studies. For example, it has been demonstrated that JSON schemas can be used to automatically derive test-case generators for web forms. In this work, we present a test-case generation approach for a rule engine-driven web-service application. Business-rule models serve us as input for property-based testing. We parse these models to automatically derive generators for sequences of web-service requests together with their required form data. Property-based testing is mostly applied in the context of functional programming. Here, we define our properties in an object-oriented style in C# and its tool FsCheck. We apply our method to the business-rule models of an industrial web-service application in the automotive domain.
A new publication has been accepted at the ICTSS 2017.
Richard Schumi, Priska Lang, Bernhard K. Aichernig, Willibald Krenn, and Rupert Schlick: “Checking Response-Time Properties of Web-Service Applications Under Stochastic User Profiles“, In Testing Software and Systems – 29th IFIP WG 6.1 International Conference, (ICTSS 2017), St. Petersburg, Russia, October 9-11, 2017, Proceedings, volume 10533 of Lecture Notes in Computer Science. Springer, 2017. (PDF)
The paper will be presented at the 29th IFIP International Conference on Testing, Software and Systems (ICTSS 2017) in St-Petersburg, Russia (9-11 October 2017).
Performance evaluation of critical software is important but also computationally expensive. It usually involves sophisticated load-testing tools and demands a large amount of computing resources. Analysing different user populations requires even more effort, becoming infeasible in most realistic cases. Therefore, we propose a model-based approach. We apply model-based test-case generation to generate log-data and learn the associated distributions of response times. These distributions are added to the behavioural models on which we perform statistical model checking (SMC) in order to assess the probabilities of the required response times. Then, we apply classical hypothesis testing to evaluate if an implementation of the behavioural model conforms to these timing requirements. This is the first model-based approach for performance evaluation combining automated test-case generation, cost learning and SMC for real applications. We realised this method with a property-based testing tool, extended with SMC functionality, and evaluate it on an industrial web-service application.
Another publication has been accepted at A-MOST 2017.
Bernhard K. Aichernig, Silvio Marcovic and Richard Schumi: “Property-Based Testing with External Test-Case Generators“, In IEEE 10th International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), 13th Workshop on Advances in Model Based Testing (A-MOST 2017), Tokyo, Japan, 13-17 March, 2017, pages 337–346. IEEE Computer Society, 2017. (PDF)(doi:10.1109/ICSTW.2017.62)
The paper will be presented on the 13th Workshop on Advances in Model Based Testing A-MOST 2017 in Tokyo, Japan on 17 March 2016. The workshop is part of the 10th IEEE International Conference on Software Testing, Verification and Validation (ICST 2017).
Previous work has demonstrated that property-based testing (PBT) is a flexible random testing technique that facilitates the generation of complex form data. For example, it has been shown that PBT can be applied to web-service applications that require various inputs for web-forms. We want to exploit this data generation feature of PBT and combine it with an external test-case generator that can generate test cases via model-based mutation testing. PBT already supports the generation of test cases from stateful models, but it is limited, because it normally only considers the current state during exploration of the model. We want to give the tester more control on how to produce meaningful operation sequences for test cases. By integrating an external test-case generator into a PBT tool, we can create test cases that follow certain coverage criteria. This allows us to reduce the test execution time, because we do not need a large number of random tests to cover certain model aspects. We demonstrate our approach with a simple example of an external generator for regular expressions and perform an industrial case study, where we integrate an existing model-based mutation testing generator.
A new publication has been accepted at the ICST 2017.
Bernhard K. Aichernig and Richard Schumi: “Statistical Model Checking Meets Property-Based Testing“, In 10th IEEE International Conference on Software Testing, Verification and Validation (ICST 2017), Tokyo, Japan, 13-17 March, 2017, pages 390-400. IEEE Computer Society, 2017. (PDF)(doi:10.1109/ICST.2017.42)
The paper will be presented at the 10th IEEE International Conference on Software Testing, Verification and Validation (ICST 2017) in Tokyo, Japan (13-17 March 2017).
In recent years, statistical model checking (SMC) has become increasingly popular, because it scales well to larger stochastic models and is relatively simple to implement. SMC solves the model checking problem by simulating the model for finitely many executions and uses hypothesis testing to infer if the samples provide statistical evidence for or against a property. Being based on simulation and statistics, SMC avoids the state-space explosion problem well-known from other model checking algorithms. In this paper we show how SMC can be easily integrated into a property-based testing framework, like FsCheck for C#. As a result we obtain a very flexible testing and simulation environment, where a programmer can define models and properties in a familiar programming language. The advantages: no external modelling language is needed and both stochastic models and implementations can be checked. In addition, we have access to the powerful test-data generators of a property-based testing tool. We demonstrate the feasibility of our approach by repeating three experiments from the SMC literature.
Another publication has been accepted at DECOSYS 2016.
Christian Burghard, Gerald Stieglbauer and Robert Korošec: “Introducing MDML – A Domain-specific Modelling Language for Automotive Measurement Devices”, In Workshop on Digital Eco-Systems, DECOSYS 2016, CEUR Workshop Proceedings, Graz, Austria 18th Oct 2016. (PDF)
The Workshop on Digital Eco-Systems DECOSYS 2016 is held at ICTSS 2016.
Another publication has been accepted at MEMOCODE’16.
Bernhard K. Aichernig and Richard Schumi: “Towards Integrating Statistical Model Checking into Property-Based Testing”. In 14th ACM/IEEE International Conference on Formal Methods and Models for System Design, MEMOCODE, Kanpur, India, Nov. 18–20, 2016, pages 71–76. IEEE Computer Society, 2016. (PDF)(doi:10.1109/MEMCOD.2016.7797748)
In recent years statistical model checking (SMC) became increasingly popular, mainly because it does not suffer from one of the major problems that limits traditional model checking, the so called state-space-explosion problem. SMC solves this problem by simulating a stochastic model for finitely many executions. There exist a number of SMC tools, but they require the user to learn a specific modelling language and a particular (temporal) logic to express properties. In this paper we propose a more flexible application of SMC, where both the model and the properties can be defined in a programming language. The technique builds upon the well-known property-based testing approach. We use the programming language C# and its associated tool FsCheck to demonstrate our approach. A stochastic counter serves as illustrating example.