Genomic selection has emerged as an effective approach in dairy cattle breeding, in which the key is prediction of genetic merit using dense SNP genotypes, i.e., genomic prediction. To improve the accuracy of genomic prediction, we need better understanding of the genetic architecture of complex traits and more sophisticated statistical modeling. In this dissertation, I developed several computing tools and performed a series of studies to investigate the genetic architecture of complex traits in dairy cattle and to improve genomic prediction models. First, we dissected additive, dominance, and imprinting effects for production, reproduction and health traits in dairy cattle. We found that non-additive effects contributed a non-negligible amount (more for reproduction traits) to the total genetic variance of complex traits in cattle. We also identified a dominant quantitative trait locus (QTL) for milk yield, revealing that detection of QTLs with non-additive effect is possible in genome-wide association studies (GWAS) using a large dataset. Second, we developed a powerful Bayesian method and a fast software tool (BFMAP) for SNP-set association and fine-mapping. We demonstrated that BFMAP achieves a power similar to or higher than existing software tools but is at least a few times faster for association tests. We also showed that BFMAP performs well for fine-mapping and can efficiently integrate fine-mapping with functional enrichment analysis. Third, we performed large-scale GWAS and fine-mapped 35 production, reproduction, and body conformation traits to single-gene resolution. We identified many novel association signals and many promising candidate genes. We also characterized causal effect enrichment patterns for a few functional annotations in dairy cattle genome and showed that our fine-mapping result can be readily used for future functional studies. Fourth, we developed an efficient Bayesian method and a fast computing tool (SSGP) for using functional annotations in genomic prediction. We demonstrated that the method and software have great potential to increase accuracy in genomic prediction and the capability to handle very large data. Collectively, these studies advance our understanding of the genetic architecture of complex traits in dairy cattle and provide fast computing tools for analyzing complex traits and improving genomic prediction.