In mammals, chromatin organization undergoes drastic reprogramming after fertilization. However, the three-dimensional structure of chromatin and its reprogramming in preimplantation development remain poorly understood. Here, by developing a low-input Hi-C (genome-wide chromosome conformation capture) approach, we examined the reprogramming of chromatin organization during early development in mice. We found that oocytes in metaphase II show homogeneous chromatin folding that lacks detectable topologically associating domains (TADs) and chromatin compartments. Strikingly, chromatin shows greatly diminished higher-order structure after fertilization. Unexpectedly, the subsequent establishment of chromatin organization is a prolonged process that extends through preimplantation development, as characterized by slow consolidation of TADs and segregation of chromatin compartments. The two sets of parental chromosomes are spatially separated from each other and display distinct compartmentalization in zygotes. Such allele separation and allelic compartmentalization can be found as late as the 8-cell stage. Finally, we show that chromatin compaction in preimplantation embryos can partially proceed in the absence of zygotic transcription and is a multi-level hierarchical process. Taken together, our data suggest that chromatin may exist in a markedly relaxed state after fertilization, followed by progressive maturation of higher-order chromatin architecture during early development.