fix(encoding): reject bytepacking overflow (#8601)