Fix test flakiness caused by slow callback duration logging