[Fiber] Detach Fragment refs during the mutation phase (#37326)