What happens if you write buggy code and misconfigure the experimental setup when evaluating a fuzzer’s performance? Wrong and misleading conclusion! We found several fatal bugs and wrong experimental settings in MLFuzz (arxiv.org/pdf/2309.16618, a revisit work on NEUZZ published on a top tier software engineering conference ASE 2023, @AndreasZeller, @ASE_conf ). These following bugs lead to wrong and misleading conclusions in MLFuzz. • An initialization bug ⇒ Failure setup of persistent mode fuzzing. • A program crash ⇒ Unexpected early termination of NEUZZ. • An error in training dataset collection ⇒ A poorly-trained neural network model. • An error in result collection ⇒ Incomplete code coverage report   We confirmed these bugs with the MLFuzz’s authors and write a rebuttal paper(arxiv.org/pdf/2409.04504) to explain the errors in MLFuzz and summarize the lessons on a fair and scientific fuzzing experiment/revisit. 1. Ensure the correctness of code implementation. Careful and rigorous debugging is needed. If you would like to patch a prior work, double-check your setting or patch is correct and seek help from original developer if needed. MLFuzz introduced 3 implementation bugs that led to wrong experimental results and conclusions.  2. Diverse benchmark selection. Try to evaluate your fuzzer on multiple benchmarks, like FuzzBench, Magma, UniFuzz. 3. Uniform code coverage metric. Covert different code coverage metrics like AFL XOR hash, LLVM coverage sanitizer (pruned), LLVM coverage sanitizer (no-pruned), AFL++ code coverage into a uniform one by replaying 4. Complete test case collection. Be sure to collect all the test cases generated by the fuzzer. 5. Uniform fuzzing mode. Ensure all fuzzer are running under same modes, either the default mode or the faster persistent mode. An apple-to-banana comparison like MLFuzz only leads to wrong conclusions. 6. Open-source your fuzzing corpus. Fuzzing is an optimization and different seed corpus (starting point) can lead to drastically variant results.

Andreas, you are a renowned researcher in the fuzzing community, and your fuzzing book is amazing. But this work draws a completely WRONG conclusion due to the careless comparison of file-retrieval fuzzer against in-memory fuzzer, where the fuzzing throughput gap is up to 10X