Decarbonization may require the efficient electrification of applications, like natural gas furnaces for space heating, that currently involve burning fossil fuels. The environmental consequences of such a fuel switch depend on location and time. Here we show that, in most parts of the U.S., a switch from natural gas to electric heat pumps would raise household heating bills and increase damages from carbon dioxide (CO2) and other pollutants. For single-family homes in 658 of the 883 locations we analyze, a shift to heat pumps would increase annual hourly peak demand for electricity. For 265 of these 658 locations peak demand already occurred in winter; for the remaining 393 heat pumps would shift peak demand from summer to winter. An economy-wide tax on CO2 emissions designed to substantially decarbonize electricity generation would make a switch to heat pumps beneficial for the environment and make electrified heating cheaper than burning natural gas. However, a sufficiently large tax would increase heating bills in cold parts of the country much more than in places with more moderate climates, even while making electric heat pumps cheaper to operate than natural gas furnaces. This differential impact raises questions about the feasibility of a carbon tax that is returned to each citizen as a uniform dividend.