Numerous studies have reported the existence of sponge-specific 16S ribosomal RNA (rRNA) gene sequence clusters, representing bacteria found in sponges but not detected in other environments, such as seawater. The advent of deep-sequencing technologies allows us to examine the rare microbial biosphere in order to establish whether these bacteria are truly sponge specific, or are more widely distributed but only at abundances below the detection limit of conventional molecular approaches. We screened >12 million publicly available 16S rRNA gene pyrotags derived from 649 seawater, sediment, hydrothermal vent and coral samples from temperate, tropical and polar regions. We detected 77 of the 173 previously described sponge-specific clusters in seawater or other non-sponge samples, albeit generally at extremely low abundances. Sequences representing the candidate phylum 'Poribacteria', previously thought to be largely restricted to sponges, were recovered from 46 (out of 411) seawater and 41 (out of 129) sediment samples. While the presence of an organism does not imply that it is active in situ, our results do suggest that many 'sponge-specific' bacteria occur more widely outside of sponge hosts than previously thought.