A meta-analysis was performed on 5 randomized controlled trials comparing surgical with non-surgical treatment for periodontal disease. The specific procedures considered were the modified Widman flap compared with scaling and root planning or curettage with anesthesia. We chose the most consistently reported outcomes, pocket depth and attachment level, for analysis. At 1 year of follow-up, surgical treatment reduced pocket depth more than non-surgical for all initial levels of disease, but by 5 years, only the deepest initial pockets (> 7 mm) showed significant improvement over non-surgically treated teeth (0.51 mm reduction, p < 0.01). Attachment level showed significantly better early results for non-surgical treatment for less diseased teeth, but by 5 years, all significant differences had disappeared. We computed quality scores following a method described by Chalmers. The mean quality score for study data analysis and presentation was 0.37 +/- 0.009 and for the study protocol, the mean quality score was 0.19 +/- 0.002. We find that this meta-analysis supports findings relating response to therapy with initial level of disease severity. We also find that the choice of outcome measure influences the choice of therapy, with surgical therapy providing greater benefit for probing depth and non-surgical therapy providing greater benefit for attachment level. These results must be viewed, however, in light of the low quality scores of the evaluated studies and the potential for bias due to lack of binding, the small mean treatment differences, and the observer measurement variability.