In recent years, deep neural networks revolutionized many aspects of computer vision. However, their success relies on massive high-quality annotated datasets that are costly to curate. This thesis is composed of three major parts. In Chapter 3, we use novel high dimensional visualization methods to explore connections between the loss landscape of neural networks and their intriguing ability to generalize to unseen test data. Next, in Chapter 4, we tackle a difficult computer vision task, namely the segmentation of anisotropic 3D electron microscopy image volumes. Deep neural networks tend to struggle in this scenario due to the lack of sufficient training data and the 3 dimensional nature of the images, as such we develop a novel state-of-the-art architecture and training workflow to improve the overall segmentation pipeline. Finally, in Chapter 5 we propose a novel state-of-the-art deep active learning algorithm for image classification to alleviate the costs of data annotations and allow networks to train effectively using less data.