Carbohydrate recognition is central to the biological and industrial exploitation of plant structural polysaccharides. These insoluble polymers are recalcitrant to microbial degradation, and enzymes that catalyze this process generally contain non-catalytic carbohydrate binding modules (CBMs) that potentiate activity by increasing substrate binding. Agarose, a repeat of the disaccharide 3,6-anhydro-alpha-L-galactose-(1,3)-beta-D-galactopyranose-(1,4), is the dominant matrix polysaccharide in marine algae, yet the role of CBMs in the hydrolysis of this important polymer has not previously been explored. Here we show that family 6 CBMs, present in two different beta-agarases, bind specifically to the non-reducing end of agarose chains, recognizing only the first repeat of the disaccharide. The crystal structure of one of these modules Aga16B-CBM6-2, in complex with neoagarohexaose, reveals the mechanism by which the protein displays exquisite specificity, targeting the equatorial O4 and the axial O3 of the anhydro-L-galactose. Targeting of the CBM6 to the non-reducing end of agarose chains may direct the appended catalytic modules to areas of the plant cell wall attacked by beta-agarases where the matrix polysaccharide is likely to be more amenable to further enzymic hydrolysis.