Glossy and matte objects can be differentiated using specular highlights: bright patches in the retinal image produced when light rays are reflected regularly from smooth surfaces. However, bright patches also occur on matte objects, due to local illumination or reflectance changes. Binocular vision provides information that could distinguish specular highlights from other luminance discontinuities; unlike surface markings, specular highlights lie not at the surface depth, but "float" in front of concave surfaces and behind convex ones. We ask whether observers implicitly understand and exploit the peculiarities of specular geometry for gloss and shape perception. Our participants judged the glossiness and shape of curved surfaces that included specular highlights at various depths. Observers demonstrated substantial deviations from a full geometric model of specular reflection. Concave surfaces appeared glossy both when highlights lay in front of and (incorrectly) behind the surface. Failings in the interpretation of monocular highlights were also apparent. Highlight disparity had no effect on shape perception. However, the perceived gloss of convex surfaces did follow geometric constraints: only highlights at appropriate depths produced high gloss ratings. We suggest, in contrast with previous work, that the visual system invokes simple heuristics as gloss indicators to accommodate complex reflections and inter-reflections that occur particularly inside concavities.