Fix observers not removing themselves from ObservedBy. (#24802)