Data visualizations allow analysts to quickly understand data trends, outliers, and patterns. However, designing the "best" visualizations for a given dataset is complicated. Multiple factors need to be considered, such as the data size, data types, target analysis tasks being supported, and even how the visualization needs to be personalized to the audience. In response, many visualization recommendation algorithms are being proposed to reduce user effort by automatically making some or all of these design decisions for analysts. However, existing visualization recommendation algorithms are evaluated in isolation, or the comparisons do not measure user performance. In other words, existing algorithms are not tested in a way that aligns with how they are used in practice. The lack of evaluation approaches makes it impossible to know how functional an algorithm is compared to another across various analysis tasks, hindering our ability to design new algorithms that provide significantly more benefits than the existing ones.This dissertation contributes a multi-faceted approach for evaluating visualization recommendation algorithms to investigate factors affecting an algorithm's performance and ways to improve it. It first proposes an evaluation-focused framework and then demonstrates how the framework can evaluate strategic behaviors and user performance among a broad range of existing algorithms. The case study results show that newly proposed algorithms might not significantly improve user performance. One way to improve the algorithm performance is by integrating more established theoretical rules or empirical results on how people perceive different visualization designs, i.e., graphical perception guidelines, to guide the recommendation ranking process. Thus, this dissertation next presents a thorough literature review of existing graphical perception literature that can inform visualization recommendation algorithms. It contributes a systematic dataset that collates existing theoretical and experimental visualization comparison results and summarizes key study outcomes. Further, this dissertation conducts an exploratory analysis to investigate the influence of each piece of graphical perception study in changing a visualization recommendation algorithm's behavior and outputs. The analysis results show that some graphical perception studies can alter the behavior of visualization recommendation algorithms dominantly, while others have little influence. Based on the analysis findings, this dissertation opens avenues at the intersection of graphical perception and visualization research, like how to evaluate the effectiveness of new graphical perception work in guiding visualization recommendations.