Skip loading encodings if quantizer is frozen