There is growing interest in understanding how diet affects the intestinal microbiota, including its possible associations with systemic diseases such as metabolic syndrome. Here we report a comprehensive and deep microbiota analysis of 14 obese males consuming fully controlled diets supplemented with resistant starch (RS) or non-starch polysaccharides (NSPs) and a weight-loss (WL) diet. We analyzed the composition, diversity and dynamics of the fecal microbiota on each dietary regime by phylogenetic microarray and quantitative PCR (qPCR) analysis. In addition, we analyzed fecal short chain fatty acids (SCFAs) as a proxy of colonic fermentation, and indices of insulin sensitivity from blood samples. The diet explained around 10% of the total variance in microbiota composition, which was substantially less than the inter-individual variance. Yet, each of the study diets induced clear and distinct changes in the microbiota. Multiple Ruminococcaceae phylotypes increased on the RS diet, whereas mostly Lachnospiraceae phylotypes increased on the NSP diet. Bifidobacteria decreased significantly on the WL diet. The RS diet decreased the diversity of the microbiota significantly. The total 16S ribosomal RNA gene signal estimated by qPCR correlated positively with the three major SCFAs, while the amount of propionate specifically correlated with the Bacteroidetes. The dietary responsiveness of the individual's microbiota varied substantially and associated inversely with its diversity, suggesting that individuals can be stratified into responders and non-responders based on the features of their intestinal microbiota.