Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Interactive Search-Based Software Testing: Development, Evaluation, and Deployment
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering. (SERL-Sweden)
2017 (English)Doctoral thesis, comprehensive summary (Other academic)
Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2017. , 235 p.
Series
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 8
Keyword [en]
Interactive Search-Based Software Testing, Search-Based Software Testing, Search-Based Software Engineering, Industrial Validation
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:bth-15083ISBN: 978-91-7295-341-3 (print)OAI: oai:DiVA.org:bth-15083DiVA: diva2:1137282
Public defence
2017-10-04, G340, Karlskrona, 13:15 (English)
Opponent
Supervisors
Available from: 2017-08-31 Created: 2017-08-30 Last updated: 2017-09-21Bibliographically approved
List of papers
1. A concept for an interactive search-based software testing system
Open this publication in new window or tab >>A concept for an interactive search-based software testing system
2012 (English)In: Lecture Notes in Computer Science, Springer , 2012, Vol. 7515, 273-278 p.Conference paper, Published paper (Refereed) Published
Abstract [en]

Software is an increasingly important part of various products, although not always the dominant component. For these software-intensive systems it is common that the software is assembled, and sometimes even developed, by domain specialists rather than by software engineers. To leverage the domain specialists' knowledge while maintaining quality we need testing tools that require only limited knowledge of software testing. Since each domain has unique quality criteria and trade-offs and there is a large variation in both software modeling and implementation syntax as well as semantics it is not easy to envisage general software engineering support for testing tasks. Particularly not since such support must allow interaction between the domain specialists and the testing system for iterative development. In this paper we argue that search-based software testing can provide this type of general and interactive testing support and describe a proof of concept system to support this argument. The system separates the software engineering concerns from the domain concerns and allows domain specialists to interact with the system in order to select the quality criteria being used to determine the fitness of potential solutions.

Place, publisher, year, edition, pages
Springer, 2012
Keyword
interactive search-based software engineering, search-based software testing, user centered
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-7104 (URN)10.1007/978-3-642-33119-0_21 (DOI)oai:bth.se:forskinfo26CA988ACD09E49CC1257AC5004BF752 (Local ID)oai:bth.se:forskinfo26CA988ACD09E49CC1257AC5004BF752 (Archive number)oai:bth.se:forskinfo26CA988ACD09E49CC1257AC5004BF752 (OAI)
Conference
4th International Symposium on Search Based Software Engineering, SSBSE
Available from: 2012-12-10 Created: 2012-11-29 Last updated: 2017-08-30Bibliographically approved
2. Objective Re-Weighting to Guide an Interactive Search Based Software Testing System
Open this publication in new window or tab >>Objective Re-Weighting to Guide an Interactive Search Based Software Testing System
2013 (English)Conference paper, Published paper (Refereed) Published
Abstract [en]

Even hardware-focused industries today develop products where software is both a large and important component. Engineers tasked with developing and integrating these products do not always have a software engineering background. To ensure quality, tools are needed that automate and support software testing while allowing these domain specialists to leverage their knowledge and experience. Search-based testing could be a key aspect in creating an automated tool for supporting testing activities. However, domain specific quality criteria and trade-offs make it difficult to develop a general fitness function a priori, so interaction between domain specialists and such a tool would be critical to its success. In this paper we present a system for interactive search-based software testing and investigate a way for domain specialists to guide the search by dynamically re-weighting quality goals. Our empirical investigation shows that objective reweighting can help a human domain specialist interactively guide the search, without requiring specialized knowledge of the system and without sacrificing population diversity.

Place, publisher, year, edition, pages
Miami: IEEE, 2013
Keyword
search based software testing, interactive search based software engineering, user centered, embedded software, industrial experience
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-6354 (URN)10.1109/ICMLA.2013.113 (DOI)000353638700017 ()oai:bth.se:forskinfo3A5EEB62A60C7D4CC1257E50004D04D9 (Local ID)978-0-7695-5144-9 (ISBN)oai:bth.se:forskinfo3A5EEB62A60C7D4CC1257E50004D04D9 (Archive number)oai:bth.se:forskinfo3A5EEB62A60C7D4CC1257E50004D04D9 (OAI)
Conference
12th International Conference on Machine Learning and Applications (ICMLA)
Available from: 2015-05-25 Created: 2015-05-25 Last updated: 2017-08-30Bibliographically approved
3. Practitioner-Oriented Visualization in an Interactive Search-Based Software Test Creation Tool
Open this publication in new window or tab >>Practitioner-Oriented Visualization in an Interactive Search-Based Software Test Creation Tool
2013 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Search-based software testing uses meta-heuristic search techniques to automate or partially automate testing tasks, such as test case generation or test data generation. It uses a fitness function to encode the quality characteristics that are relevant, for a given problem, and guides the search to acceptable solutions in a potentially vast search space. From an industrial perspective, this opens up the possibility of generating and evaluating lots of test cases without raising costs to unacceptable levels. First, however, the applicability of search-based software engineering in an industrial setting must be evaluated. In practice, it is difficult to develop a priori a fitness function that covers all practical aspects of a problem. Interaction with human experts offers access to experience that is otherwise unavailable and allows the creation of a more informed and accurate fitness function. Moreover, our industrial partner has already expressed a view that the knowledge and experience of domain specialists are more important to the overall quality of the systems they develop than software engineering expertise. In this paper we describe our application of Interactive Search Based Software Testing (ISBST) in an industrial setting. We used SBST to search for test cases for an industrial software module and based, in part, on interaction with a human domain specialist. Our evaluation showed that such an approach is feasible, though it also identified potential difficulties relating to the interaction between the domain specialist and the system.

Place, publisher, year, edition, pages
IEEE Computer Society, 2013
Keyword
interactive systems, program testing, ISBST, industrial software module, interactive search-based software test creation tool, meta-heuristic search techniques, practitioner-oriented visualization
National Category
Computer Systems
Identifiers
urn:nbn:se:bth-11366 (URN)10.1109/APSEC.2013.120 (DOI)
Conference
20th Asia - Pacific Software Engineering Conference (APSEC), Bangkok
Funder
Knowledge Foundation, 2010/0124
Available from: 2016-01-05 Created: 2016-01-05 Last updated: 2017-08-30Bibliographically approved
4. An initial industrial evaluation of interactive search-based testing for embedded software
Open this publication in new window or tab >>An initial industrial evaluation of interactive search-based testing for embedded software
2015 (English)In: Applied Soft Computing, ISSN 1568-4946, E-ISSN 1872-9681, Vol. 29, 26-39 p.Article in journal (Refereed) Published
Abstract [en]

Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. From an industrial perspective, this could enable an increase in product quality without a matching increase in the time and effort required to do so. Search-based software testing, however, is a set of quite complex techniques and approaches that do not immediately translate into a process for use with most companies. For example, even if engineers receive the proper education and training in these new approaches, it can be hard to develop a general fitness function that covers all contingencies. Furthermore, in industrial practice, the knowledge and experience of domain specialists are often key for effective testing and thus for the overall quality of the final software system. But it is not clear how such domain expertise can be utilized in a search-based system. This paper presents an interactive search-based software testing (ISBST) system designed to operate in an industrial setting and with the explicit aim of requiring only limited expertise in software testing. It uses SBST to search for test cases for an industrial software module, while also allowing domain specialists to use their experience and intuition to interactively guide the search. In addition to presenting the system, this paper reports on an evaluation of the system in a company developing a framework for embedded software controllers. A sequence of workshops provided regular feedback and validation for the design and improvement of the ISBST system. Once developed, the ISBST system was evaluated by four electrical and system engineers from the company (the ’domain specialists’ in this context) used the system to develop test cases for a commonly used controller module. As well as evaluating the utility of the ISBST system, the study generated interaction data that were used in subsequent laboratory experimentation to validate the underlying search-based algorithm in the presence of realistic, but repeatable, interactions. The results validate the importance that automated software testing tools in general, and search-based tools, in particular, can leverage input from domain specialists while generating tests. Furthermore, the evaluation highlighted benefits of using such an approach to explore areas that the current testing practices do not cover or cover insufficiently. © 2014 Elsevier B.V. All rights reserved.

Keyword
Ability testing; Automatic test pattern generation; Embedded software; Personnel training; Search engines; Software engineering, Automated software testing; Education and training; Industrial experience; Interactive search; Knowledge and experience; Search-based algorithms; Search-based software testing; User-centered, Software testing
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-718 (URN)10.1016/j.asoc.2014.12.025 (DOI)000350648100003 ()2-s2.0-84920672523 (Scopus ID)
Available from: 2015-06-01 Created: 2015-05-28 Last updated: 2017-08-30Bibliographically approved
5. Tester interactivity makes a difference in search-based software testing: A controlled experiment
Open this publication in new window or tab >>Tester interactivity makes a difference in search-based software testing: A controlled experiment
Show others...
2016 (English)In: Information and Software Technology, ISSN 0950-5849, E-ISSN 1873-6025, Vol. 78, 66-82 p.Article in journal (Refereed) Published
Abstract [en]

Context: Search-based software testing promises to provide users with the ability to generate high quality test cases, and hence increase product quality, with a minimal increase in the time and effort required. The development of the Interactive Search-Based Software Testing (ISBST) system was motivated by a previous study to investigate the application of search-based software testing (SBST) in an industrial setting. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants. Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system. Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment. Results:The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results. In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance. Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result. (C) 2016 Elsevier B.V. All rights reserved.

Place, publisher, year, edition, pages
Elsevier, 2016
Keyword
Search-based software testing, Interactive search-based software testing, Controlled experiment
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-12958 (URN)10.1016/j.infsof.2016.05.009 (DOI)000379634100004 ()
External cooperation:
Available from: 2016-08-31 Created: 2016-08-30 Last updated: 2017-08-30Bibliographically approved
6. Using Exploration Focused Techniques to Augment Search-Based Software Testing: An Experimental Evaluation
Open this publication in new window or tab >>Using Exploration Focused Techniques to Augment Search-Based Software Testing: An Experimental Evaluation
2016 (English)In: Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, IEEE Computer Society, 2016, 69-79 p., 7515460Conference paper, Published paper (Refereed)
Abstract [en]

Search-based software testing (SBST) often uses objective-based approaches to solve testing problems. There are, however, situations where the validity and completeness of objectives cannot be ascertained, or where there is insufficient information to define objectives at all. Incomplete or incorrect objectives may steer the search away from interesting behavior of the software under test (SUT) and from potentially useful test cases. This papers investigates the degree to which exploration-based algorithms can be used to complement an objective-based tool we have previously developed and evaluated in industry. In particular, we would like to assess how exploration-based algorithms perform in situations where little information on the behavior space is available a priori. We have conducted an experiment comparing the performance of an exploration-based algorithm with an objective-based one on a problem with a high-dimensional behavior space. In addition, we evaluate to what extent that performance degrades in situations where computational resources are limited. Our experiment shows that exploration-based algorithms are useful in covering a larger area of the behavior space and result in a more diverse solution population. Typically, of the candidate solutions that exploration-based algorithms propose, more than 80% were not covered by their objective-based counterpart. This increased diversity is present in the resulting population even when computational resources are limited. We conclude that exploration-focused algorithms are a useful means of investigating high-dimensional spaces, even in situations where limited information and limited resources are available.

Place, publisher, year, edition, pages
IEEE Computer Society, 2016
Series
IEEE International Conference on Software Testing Verification and Validation, ISSN 2381-2834
Keyword
Algorithms; Verification, Computational resources; Controlled experiment; Diverse solutions; Experimental evaluation; High dimensional spaces; High-dimensional; Limited information; Search-based software testing, Software testing
National Category
Software Engineering
Identifiers
urn:nbn:se:bth-13094 (URN)10.1109/ICST.2016.26 (DOI)000391252900007 ()2-s2.0-84983246859 (Scopus ID)9780306406157 (ISBN)
Conference
9th IEEE International Conference on Software Testing, Verification and Validation, ICST 2016; Chicago
Available from: 2016-10-06 Created: 2016-10-03 Last updated: 2017-08-30Bibliographically approved
7. Transferring Interactive Search-Based Software Testing to Industry
Open this publication in new window or tab >>Transferring Interactive Search-Based Software Testing to Industry
(English)Manuscript (preprint) (Other academic)
Keyword
interactive search-based software testing, industrial evaluation
National Category
Computer Systems
Identifiers
urn:nbn:se:bth-15082 (URN)
Available from: 2017-08-30 Created: 2017-08-30 Last updated: 2017-08-31Bibliographically approved

Open Access in DiVA

fulltext(5012 kB)0 downloads
File information
File name FULLTEXT01.pdfFile size 5012 kBChecksum SHA-512
e8e94d7f1c230dda1d3c32390f4f835524b962eb4c641612c1a9bd4a33c7c0348119c57dbad9ff119e683c14c8ddf3239dd7bcb904f8deb828433e517bfadb79
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Marculescu, Bogdan
By organisation
Department of Software Engineering
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 183 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf