Recent years have seen huge interest in understanding how demographic factors mediate the evolution of social behavior in viscous populations. Here we study the impact of variation in group size on the evolution of helping and harming behavior. Although variation in group size influences the degree of relatedness and the degree of competition between groupmates, we find that these effects often exactly cancel, so as to give no net impact of variation in group size on the evolution of helping and harming. Specifically, (1) obligate helping and harming are never mediated by variation in group size, (2) facultative helping and harming are not mediated by variation in group size when this variation is spatial only, (3) facultative helping and harming are mediated by variation in group size only when this variation is temporal or both spatial and temporal, and (4) when there is an effect of variation in group size, facultative helping is favored in big groups and facultative harming is favored in little groups. Moreover, we find that spatial and temporal heterogeneity in individual fecundity may interact with patch-size heterogeneity to change these predictions, promoting the evolution of harming in big patches and of helping in little patches.