Colloidal lithography utilizes self-assembled particle monolayers as lithographic masks to fabricate arrays of nanostructures by combination of directed evaporation and etching steps. This process provides complex nanostructures over macroscopic areas in a simple, convenient, and parallel fashion without requiring clean-room infrastructure and specialized equipment. The appeal of the method comes at the price of imperfections impairing the optical quality, especially for arrayed nanostructures relying on well-ordered lattices. Imperfections are often generically mentioned to rationalize the discrepancy between experimental and simulated resonances. Yet, little attention is given to detailed structure-property relationships connecting typical defects directly with the optical properties. Here, we use a correlative approach to connect nano- and microscopic defects occurring from the colloidal lithography process with the resulting local optical properties. We use nanohole arrays as a common plasmonic structure known to be sensitive to lattice imperfections. Correlative optical and electron microscopies reveal the individual role of packing order, organic impurities, and solid polymer bridges. Our findings show that simple cleaning processes with solvents and oxygen plasma already improve the optical quality but also highlight how well-controlled self-assembly processes are required for predictable optical properties of such nanostructures.