MINOR: Speed up AuthorizerIntegrationTest (#22785)