NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentation

Despite recent advances in Open-Vocabulary Semantic Segmentation (OVSS), existing training-free methods face several limitations: use of computationally expensive affinity refinement strategies, ineffective fusion of trans- former attention maps due to equal weighting or reliance on fixed-size Gaussian kernels to reinforce local spatial smoothness, enforcing isotropic neighborhoods. We pro- pose a strong baseline for training-free OVSS termed as NERVE (Neighbourhood & Entropy-guided Random-walk for open-Vocabulary sEgmentation), which uniquely inte- grates global and fine-grained local information, exploiting the neighbourhood structure from the self-attention layer of a stable diffusion model. We also introduce a stochas- tic random walk for refining the affinity rather than relying on fixed-size Gaussian kernels for local context. This spa- tial diffusion process encourages propagation across con- nected and semantically related areas, enabling it to ef- fectively delineate objects with arbitrary shapes. Whereas most existing approaches treat self-attention maps from dif- ferent transformer heads or layers equally, our method uses entropy-based uncertainty to select the most relevant maps. Notably, our method does not require any conven- tional post-processing techniques like Conditional Random Fields (CRF) or Pixel-Adaptive Mask Refinement (PAMR). Experiments are performed on 7 popular semantic segmen- tation benchmarks, yielding an overall state-of-the-art zero- shot segmentation performance, providing an effective ap- proach to open-vocabulary semantic segmentation.