How_to_Evaluate_Solutions_in_Pareto-Based_Search-Based_Software_Engineering_A_Critical_Review_and_Methodological_Guidance.pdf (6.54 MB)
Download file

How to evaluate solutions in Pareto-based search-based software engineering: a critical review and methodological guidance

Download (6.54 MB)
journal contribution
posted on 19.02.2021, 10:55 by Miqing Li, Tao ChenTao Chen, Xin Yao

With modern requirements, there is an increasing tendency of considering multiple objectives/criteria simultaneously in many Software Engineering (SE) scenarios. Such a multi-objective optimization scenario comes with an important issue - how to evaluate the outcome of optimization algorithms, which typically is a set of incomparable solutions (i.e., being Pareto nondominated to each other). This issue can be challenging for the SE community, particularly for practitioners of Search-Based SE (SBSE). On one hand, multi-objective optimization could still be relatively new to SE/SBSE researchers, who may not be able to identify the right evaluation methods for their problems. On the other hand, simply following the evaluation methods for general multi-objective optimization problems may not be appropriate for specific SBSE problems, especially when the problem nature or decision maker's preferences are explicitly/implicitly known. This has been well echoed in the literature by various inappropriate/inadequate selection and inaccurate/misleading use of evaluation methods. In this paper, we first carry out a systematic and critical review of quality evaluation for multi-objective optimization in SBSE. We survey 717 papers published between 2009 and 2019 from 36 venues in seven repositories, and select 95 prominent studies, through which we identify five important but overlooked issues in the area. We then conduct an in-depth analysis of quality evaluation indicators/methods and general situations in SBSE, which, together with the identified issues, enables us to codify a methodological guidance for selecting and using evaluation methods in different SBSE scenarios.

Funding

Guangdong Provincial Key Laboratory (Grant No. 2020B121201001)

Program for Guangdong Introducing Innovative and Enterpreneurial Teams (Grant No. 2017ZT07X386)

Shenzhen Science and Technology Program (Grant No. KQTD2016112514355531)

Program for University Key Laboratory of Guangdong Province (Grant No. 2017KSYS008)

History

School

  • Science

Department

  • Computer Science

Published in

IEEE Transactions on Software Engineering

Volume

48

Issue

5

Pages

1771 - 1799

Publisher

Institute of Electrical and Electronics Engineers (IEEE)

Version

VoR (Version of Record)

Rights holder

© The Authors

Publisher statement

This is an Open Access Article. It is published by IEEE under the Creative Commons Attribution 4.0 International Licence (CC BY). Full details of this licence are available at: https://creativecommons.org/licenses/by/4.0/

Acceptance date

17/10/2020

Publication date

2020-11-09

Copyright date

2020

ISSN

0098-5589

eISSN

1939-3520

Language

en

Depositor

Dr Tao Chen Deposit date: 18 February 2021