The early diagnosis of Alzheimer's disease (AD) is crucial because individuals may first experience mild cognitive impairment (MCI), which can then develop into AD, enabling timely intervention, slowing disease progression, and advancing the understanding of AD pathology. However, existing methods face two major challenges: first, they lack effective mechanisms to handle abnormal samples in neuroimaging data, which can distort model learning; second, they do not fully exploit complementary structural information across modalities, leading to insufficient discriminative power. To tackle these problems, we propose a model for outlier detection and cross-modal representation learning. This model leverages graph fusion for effective cross-modal information utilization and introduces multiple latent space mappings. Additionally, an outlier detection vector assigns lower learning weights to more anomalous samples, mitigating their impact. An alternating optimization algorithm ensures convergence and optimizes the objective function. Experimental comparisons with related algorithms on AD datasets demonstrate our method's superiority. These results confirm that explicitly addressing abnormal data and enhancing cross-modal fusion are essential for improving both the robustness and accuracy of AD early diagnosis.