We investigate a counterintuitive geometric interaction between defects and curvature in thin layers of superfluids, superconductors, and liquid crystals deposited on curved surfaces. Each defect feels a geometric potential whose functional form is determined only by the shape of the surface, but whose sign and strength depend on the transformation properties of the order parameter. For superfluids and superconductors, the strength of this interaction is proportional to the square of the charge and causes all defects to be repelled (attracted) by regions of positive (negative) Gaussian curvature. For liquid crystals in the one elastic constant approximation, charges between 0 and 4pi are attracted by regions of positive curvature while all other charges are repelled.