Back to search

IKTPLUSS-IKT og digital innovasjon

EET: Evolutionary Enterprise Testing

Alternative title: EET: Evolusjonær Enterprise Testing

Awarded: NOK 7.8 mill.

Nowadays, software affects most parts of life, like banking, healthcare, enterprises, transportation, smartphones, entertainment systems, etc. Unfortunately, writing software is hard, and, most of the time, systems are shipped with bugs, i.e. functional mistakes. Software testing is used to try to find those bugs, but it is a complex, tedious task, especially when done manually. Often, software testing takes up to half of the development time and cost for a system. As of 2013, it is estimated that software testing is costing $312 billions worldwide. So much testing is needed because the cost of software failure is simply too large: in 2016, 548 recorded and documented software failures impacted 4.4 billion people and $1.1 trillion in assets worldwide. One of the most famous cases of software failure is the explosion, in 1996, of the Arianne 5 rocket (estimated worth $500 millions) due to a numerical bug in its controller software. Automating software testing would not only be significantly cost saving, but would also prevent severe consequences in critical systems, e.g. people have died due to software malfunctions in medical equipment, like for example when eight patients died in 2000 due to radiation overdose caused by a software bug. Software testing is a very complex task to automate, e.g., how to generate effective test cases automatically. As such, cutting-edge technology and research is needed to solve this very complex problem which impacts billions of people. Search-based approaches that rely on evolutionary computation algorithms are among the most promising techniques to tackle this type of hard problem. However, more research is needed to make them more effective and scalable to the complexity of large, modern software systems. In this project, we aim at providing novel techniques based on evolutionary algorithms to solve the problem of automatically generating system-level test cases. In particular, we focus on enterprise systems. Current research prototype has been already used to automatically find several bugs in many open-source and industrial systems. It can be downloaded from www.evomaster.org. New techniques (e.g. based on SQL handling) have been developed to improve performance even further.

With 20 published articles, 12 of them at Level 2 in the Norwegian system, the project has provided significant contribution to the state-of-the-art in software testing research. The open-source tool EvoMaster (www.evomaster.org) developed in this project is already used by several companies worldwide, like large e-commerce ones such as Meituan in China, having direct impact on software engineering practice. Furthermore, several research groups outside Norway have been already using the open-source tools developed in this project for their research. The work done in this project was the essential foundation for bootstrapping new research directions, investigated in an ERC Consolidator grant, and used in European consortiums on AI research for software testing. Unfortunately, though, despite these major achievements, all efforts to continue to build on this success in Norway (ie, funding from NFR to keep working on this topic) have failed so far, limiting its further potential impact.

Nowadays, software affects most parts of life, like banking, healthcare, enterprises, transportation, smartphones, entertainment systems, etc. Unfortunately, writing software is hard, and, most of the time, systems are shipped with bugs, i.e. functional mistakes. Software testing is used to try to find those bugs, but it is a complex, tedious task, especially when done manually. Often, software testing takes up to half of the development time and cost for a system. As of 2013, it is estimated that software testing is costing $312 billions worldwide. So much testing is needed because the cost of software failure is simply too large: in 2016, 548 recorded and documented software failures impacted 4.4 billion people and $1.1 trillion in assets worldwide. One of the most famous cases of software failure is the explosion, in 1996, of the Arianne 5 rocket (estimated worth $500 millions) due to a numerical bug in its controller software. Automating software testing would not only be significantly cost saving, but would also prevent severe consequences in critical systems, e.g. people have died due to software malfunctions in medical equipment, like for example when eight patients died in 2000 due to radiation overdose caused by a software bug. Software testing is a very complex task to automate, e.g., how to generate effective test cases automatically. As such, cutting-edge technology and research is needed to solve this very complex problem which impacts billions of people. Search-based approaches that rely on evolutionary computation algorithms are among the most promising techniques to tackle this type of hard problem. However, more research is needed to make them more effective and scalable to the complexity of large, modern software systems. In this project, we aim at providing novel techniques based on evolutionary algorithms to solve the problem of automatically generating system-level test cases. In particular, we focus on enterprise systems.

Funding scheme:

IKTPLUSS-IKT og digital innovasjon