Machine Learning for Image Processing

Scene Segmentation



The goal of scene segmentation is to generate pixel-wise segmentation such that each pixel is labeled with either one of background classes (ex. sky, building, road, water, etc.) or one of foreground classes (ex. human, car, boat, bicycle, etc.).

Image partitioning is an important preprocessing step for many state-of-the-art algorithms used for solving high-level computer vision problems. Typically, partitioning is conducted without any regards to the task at hand. We believe that this task-oblivious partitioning of the image into superpixels (or regions) results in a suboptimal image representation for any particular specific image labeling task. We propose a task-specific image partitioning framework as an effective way to produce region-based image representations that are appropriate for the task at hand. We achieve this by learning an image partitioning model based on a correlation clustering objective from a task-specific training dataset. Furthermore, we propose a novel higher-order correlation clustering that considers higher-order relations among superpixels. We evaluate the learnt task-aware partitioning algorithms on the problems of semantic scene segmentation and surface layout labeling. Our key result is to show that task-aware partitioning leads to better labeling performance than the partitioning computed by state-of-the-art task-oblivious partitioning algorithms.


Fig.1 Scene Segmentation

Fig.2 Higher-order Correlation Clustering


In addition to pairwise relations between neighboring superpixels, we generalize a correlation clustering to incorporate higher-order relations over a hypergraph, which enables to capture long-range dependencies of distant nodes (superpixels) and global context. For this higher-order correlation clustering, we extend the LP relaxation approach for inference in such model and also use structured support vector machine for supervised training of task-specific parameters. Experimental results show that the proposed correlation clustering outperforms other state-of-the-art image partitioning algorithms.


Fig. 3 Qualitative Results