Germ granules are germ lineage-specific ribonucleoprotein (RNP) complexes, but how they are assembled and specifically segregated to germ lineage cells remains unclear. Here, we show that the PGL proteins PGL-1 and PGL-3 serve as the scaffold for germ granule formation in Caenorhabditis elegans. Using cultured mammalian cells, we found that PGL proteins have the ability to self-associate and recruit RNPs. Depletion of PGL proteins from early C. elegans embryos caused dispersal of other germ granule components in the cytoplasm, suggesting that PGL proteins are essential for the architecture of germ granules. Using a structure-function analysis in vivo, we found that two functional domains of PGL proteins contribute to germ granule assembly: an RGG box for recruiting RNA and RNA-binding proteins and a self-association domain for formation of globular granules. We propose that self-association of scaffold proteins that can bind to RNPs is a general mechanism by which large RNP granules are formed.