Image segmentation is a fundamental problem of low level computer vision and is also used as a preprocessing step for a number of higher level tasks (e.g. object detection and recognition, action classification, optical flow and stereo computation etc). In this dissertation we study the image segmentation problem focusing on the task of segmentation into surfaces.

First we present our unifying framework through which mean shift, bilateral filtering and anisotropic diffusion can be described. Three new methods are also described and implemented and the most prominent of them, called Color Mean Shift (CMS), is extensively tested and compared against the existing methods. We experimentally show that CMS outperforms the other methods i.e., creates more uniform regions and retains equally well the edges between segments.

Next we argue that color based segmentation should be a two stage process; edge preserving filtering, followed by pixel clustering. We create novel segmentation algorithms by coupling the previously described filtering methods with standard grouping techniques. We compare all the segmentation methods with current state of the art grouping methods and show that they produce better results on the Berkeley and Weizmann segmentation datasets. A number of other interesting conclusions are also drawn from the comparison.

Then we focus on surface normal estimation techniques. We present two novel methods to estimate the parameters of a planar surface viewed by a moving robot when the odometry is known. We also present a way of combining them and integrate the measurements over time using an extended Kalman filter. We test the estimation accuracy by demonstrating the ability of the system to navigate in an indoor environment using exclusively vision.

We conclude this dissertation with a discussion on how color based segmentation can be integrated into a structure from motion framework that computes planar surfaces using homographies.