The effects of the weak interactions of rare gas atoms on the UV-visible absorption spectra of gold dimer and tetramer clusters are investigated. The time-dependent density functional theory based on the two-component relativistic zeroth-order regular approximation that considered spin-orbit coupling is performed to estimate the absorption spectra of Au2,4-Rgn (Rg = Ne-Xe, and n = 1-6) complexes. Using spin-orbit, including the appropriate functional, shows a close correlation between experiment and our calculations. It is also demonstrated that the weak interactions between rare gas atoms and gold clusters affect the UV-vis spectra of Au2,4 clusters by shifting the electronic transition toward the blue. Moreover, we find that the order of change in peak position, Δν̃, is proportional to the strength of interactions: Δν̃Au2,4-Xe > Δν̃Au2,4-Kr > Δν̃Au2,4-Ar > Δν̃Au2,4-Ne. In addition, comparing the UV-visible spectra of Au2,4-Rgn complexes with those of isolated Au2 and Au4 clusters shows that for Au2,4-Rg2,4,6 complexes in which Rg atoms interacted symmetrically with gold clusters no additional peaks are observed compared to isolated clusters; however, for Au2,4-Rg1,3,5 complexes, extra peaks appear because of the decrease in symmetry.