Evolutionary multi-objective optimization (EMO) has been a very active research area in recent years. Almost every year, new EMO algorithms are proposed. When a new EMO algorithm is proposed, computational experiments are usually conducted in order to compare its performance with existing algorithms. Then, experimental results are summarized and reported as a number of tables together with statistical significance test results. Those results usually show higher performance of the new algorithm than existing algorithms. However, fair comparison of different EMO algorithms is not easy since the evaluated performance of each algorithm usually depends on experimental settings. This is also because solution sets instead of solutions are evaluated.
In this tutorial, we will explain and discuss various difficulties in fair performance comparison of EMO algorithms related to the following four issues: (i) the termination condition of each algorithm, (ii) the population size of each algorithm, (iii) performance indicators, (iv) test problems. For each issue, its strong effects on comparison results are clearly demonstrated. Our discussions on those difficulties are to encourage the future development of the EMO research field without excessively focusing on the proposal of overly-specialized new algorithms in a specific setting. This is because those algorithms are not likely to work well on various real-world tasks. Then, we will discuss the handling of each issue for fair comparison. We will also suggest some promising future research topics related to each issue.