Fix int8 mm being skipped on offloaded lora weights. (#14669)