Leisure-time physical activity (PA) and exercise training are known to help maintain cognitive function in healthy older adults. However, relatively little is known about the effects of PA on cognitive function or brain function in those at increased risk for Alzheimer’s disease through the presence of the apolipoproteinE epsilon4 (APOE-ε4) allele, diagnosis of mild cognitive impairment (MCI), or the presence of metabolic disease. Here, we examine the question of whether PA and exercise interventions may differentially impact cognitive trajectory, clinical outcomes, and brain structure and function among individuals at the greatest risk for AD. The literature suggests that the protective effects of PA on risk for future dementia appear to be larger in those at increased genetic risk for AD. Exercise training is also effective at helping to promote stable cognitive function in MCI patients, and greater cardiorespiratory fitness is associated with greater brain volume in early-stage AD patients. In APOE-ε4 allele carriers compared to non-carriers, greater levels of PA may be more effective in reducing amyloid burden and are associated with greater activation of semantic memory-related neural circuits. A greater research emphasis should be placed on randomized clinical trials for exercise, with clinical, behavioral, and neuroimaging outcomes in people at increased risk for AD.