We perform quantitative measurements of the actual area of contact, A, formed by two rough solids that are subjected to different normal loading protocols. We show that microscopic motion, induced by Poisson contraction or expansion, produces a strong memory dependence of on the loading history with a large corresponding influence on the system's frictional strength. These effects, together with accompanying transient dynamics, are independent of humidity, loading rates, and material contrast across the interface.