[cache] Simplify cache dispatch based on layer_types (#47118)