Land degradation is a global problem that particularly impacts the poor rural inhabitants of low and middle-income countries. We improve upon existing literature by estimating the extent of rural populations in 2000 and 2010 globally on degrading and improving agricultural land, taking into account the role of market access, and analyzing the resulting impacts on poverty. Using a variety of spatially referenced datasets, we estimate that 1.33 billion people worldwide in 2000 were located on degrading agricultural land (DAL), of which 1.26 billion were in developing countries. Almost all the world's 200 million people on remote DAL were in developing countries, which is about 6% of their rural population. There were also 1.54 billion rural people on improving agricultural land (IAL), with 1.34 billion in developing countries. We find that a lower share of people in 2000 on DAL, or a higher share on IAL, lowers significantly how much overall economic growth reduces poverty from 2000 to 2012 across 83 developing countries. As the population on DAL and IAL in developing countries grew by 13% and 15% respectively from 2000 to 2010, these changing spatial distributions of rural populations could impact significantly future poverty in developing countries.