BBSR - Benchmarking, Benchmarks, Software, and Reproducibility

Description

The Benchmarking, Benchmarks, Software, and Reproducibility track welcomes submissions that touch on all aspects of reproducibility, benchmarking, and software of genetic and evolutionary computation methods. In particular, we welcome submissions on the following topics:

• Benchmarking methodologies for assessing the performance of evolutionary algorithms and related optimization techniques,
• Benchmark problems and toolboxes for evaluating evolutionary computation methods or enabling the training of meta-learning techniques for these,
• Statistical analysis and visualization techniques for understanding problem spaces or the performance and behavior of optimization techniques, including instance space analysis and landscape analysis,
• Reproducibility studies that rigorously replicate published experiments with a substantial shift in confidence in the results of the original study,
• Innovative software for deploying, evaluating, developing, or teaching genetic and evolutionary computation in original and unique ways.

This is a non-exhaustive list, and we invite the authors to get in touch with the track chairs if in doubt about the suitability of their submission to this track.

Requirements for reproducibility studies

For reproducibility studies, the reasons for the new findings must be clearly explained in order to ensure a meaningful and distinct contribution from the original study (e.g. different benchmarks, application scenarios, technical or implementation differences). The submission must follow the highest reproducibility standards by providing all implementation details, input data, parameters and hardware specifications. All artifacts must be made available in a public repository upon submission and must remain available after publication. The submission must also follow the usual standards in terms of plagiarism.
Of particular interest are replicability studies, defined as follows by the
ACM: "The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials (different team, different experimental setup)."

Anonymization

We acknowledge that for some of the works fitting this track, it may be difficult to submit in completely anonymized form, e.g., when links to demos, data, or software are required to assess the suitability of the submission for GECCO. Whenever possible, we strongly encourage the authors to make use of anonymous repositories (available on Zenodo and for GitHub repositories, for example). In the ideal case, these repositories will be deanonymized only after the notification. Where it is impossible to anonymize repositories, the BBSR track allows to link resources that possibly reveal authors’ identity. However, also in this case, all other elements of the paper shall follow the standard anonymization guidelines. In particular, we require that author names, affiliations, and acknowledgments are suppressed and that, to the maximum extent possible, references to any of the author's own work should be made as if the work belonged to someone else. We strongly recommend the use of the following option:

\documentclass[dvipsnames,format=sigconf,anonymous=true,review=true]{acmart}