Recent advances in the field of optical microscopy have enabled scientists to

observe and image complex biological processes across a wide range of spatial and

temporal resolution, resulting in an exponential increase in optical microscopy data.

Manual analysis of such large volumes of data is extremely time consuming and often

impossible if the changes cannot be detected by the human eye. Naturally it is essential

to design robust, accurate and high performance image processing and analysis

tools to extract biologically significant results. Furthermore, the presentation of the

results to the end-user, post analysis, is also an equally challenging issue, especially

when the data (and/or the hypothesis) involves several spatial/hierarchical scales

(e.g., tissues, cells, (sub)-nuclear components). This dissertation concentrates on

a subset of such problems such as robust edge detection, automatic nuclear segmentation

and selection in multi-dimensional tissue images, spatial analysis of gene

localization within the cell nucleus, information visualization and the development

of a computational framework for efficient and high-throughput processing of large

datasets.

Initially, we have developed 2D nuclear segmentation and selection algorithms

which help in the development of an integrated approach for determining the preferential

spatial localization of certain genes within the cell nuclei which is emerging

as a promising technique for the diagnosis of breast cancer. Quantification requires

accurate segmentation of 100 to 200 cell nuclei in each patient tissue sample in order

to draw a statistically significant result. Thus, for large scale analysis involving hundreds

of patients, manual processing is too time consuming and subjective. We have

developed an integrated workflow that selects, following 2D automatic segmentation,

a sub-population of accurately delineated nuclei for positioning of fluorescence in

situ hybridization labeled genes of interest in tissue samples. Application of the

method was demonstrated for discriminating normal and cancerous breast tissue

sections based on the differential positioning of the HES5 gene. Automatic results

agreed with manual analysis in 11 out of 14 cancers, all 4 normal cases and all 5

non-cancerous breast disease cases, thus showing the accuracy and robustness of the

proposed approach.

As a natural progression from the 2D analysis algorithms to 3D, we first developed

a robust and accurate probabilistic edge detection method for 3D tissue

samples since several down stream analysis procedures such as segmentation and

tracking rely on the performance of edge detection. The method based on multiscale

and multi-orientation steps surpasses several other conventional edge detectors

in terms of its performance. Subsequently, given an appropriate edge measure, we

developed an optimal graphcut-based 3D nuclear segmentation technique for samples

where the cell nuclei are volume or surface labeled. It poses the problem as

one of finding minimal closure in a directed graph and solves it efficiently using the

maxflow-mincut algorithm. Both interactive and automatic versions of the algorithm

are developed. The algorithm outperforms, in terms of three metrics that are

commonly used to evaluate segmentation algorithms, a recently reported geodesic

distance transform-based 3D nuclear segmentation method which in turns was reported

to outperform several other popular tools that segment 3D nuclei in tissue

samples.

Finally, to apply some of the aforementioned methods to large microscopic

datasets, we have developed a user friendly computing environment called MiPipeline

which supports high throughput data analysis, data and process provenance,

visual programming and seamlessly integrated information visualization of hierarchical

biological data. The computational part of the environment is based on LONI

Pipeline distributed computing server and the interactive information visualization

makes use of several javascript based libraries to visualize an XML-based backbone

file populated with essential meta-data and results.