Sensory neurons in the mouse eye and nose have unusual chromatin organization. Here we report their three-dimensional (3D) genome structure at 20-kilobase (kb) resolution, achieved by applying our recently developed diploid chromatin conformation capture (Dip-C) method to 409 single cells from the retina and the main olfactory epithelium of adult and newborn mice. The 3D genome of rod photoreceptors exhibited inverted radial distribution of euchromatin and heterochromatin compared with that of other cell types, whose nuclear periphery is mainly heterochromatin. Such genome-wide inversion is not observed in olfactory sensory neurons (OSNs). However, OSNs exhibited an interior bias for olfactory receptor (OR) genes and enhancers, in clear contrast to non-neuronal cells. Each OSN harbored multiple aggregates of OR genes and enhancers from different chromosomes. We also observed structural heterogeneity of the protocadherin gene cluster. This type of genome organization may provide the structural basis of the 'one-neuron, one-receptor' rule of olfaction.