Just wildly guessing here -
wherever there are glossy/coated/metal materials in a scene they throw noise from reflective caustics all over the place and this is only slowly to resolve by path tracing.
Now the converging speed on a certain surface may not be correlated with the noise level - depending on what reflective item creates the caustics and where.
And when there are less noisy areas taking actually longer to converge adaptivity won't help.
I'm waiting for glossy and metal getting included in caustic cache - theoretically that might help solving this.
Blurry reflections work well with big merge radius = small caustic cache not slowing down things too much.
But maybe I'm wrong about this...