Tin and lead iodide perovskite semiconductors of the composition AMX3, where M is a metal and X is a halide, are leading candidates for high efficiency low cost tandem photovoltaics, in part because they have band gaps that can be tuned over a wide range by compositional substitution. We experimentally identify two competing mechanisms through which the A-site cation influences the band gap of 3D metal halide perovskites. Using a smaller A-site cation can distort the perovskite lattice in two distinct ways: by tilting the MX6 octahedra or by simply contracting the lattice isotropically. The former effect tends to raise the band gap, while the latter tends to decrease it. Lead iodide perovskites show an increase in band gap upon partial substitution of the larger formamidinium with the smaller cesium, due to octahedral tilting. Perovskites based on tin, which is slightly smaller than lead, show the opposite trend: they show no octahedral tilting upon Cs-substitution but only a contraction of the lattice, leading to progressive reduction of the band gap. We outline a strategy to systematically tune the band gap and valence and conduction band positions of metal halide perovskites through control of the cation composition. Using this strategy, we demonstrate solar cells that harvest light in the infrared up to 1040 nm, reaching a stabilized power conversion efficiency of 17.8%, showing promise for improvements of the bottom cell of all-perovskite tandem solar cells. The mechanisms of cation-based band gap tuning we describe are broadly applicable to 3D metal halide perovskites and will be useful in further development of perovskite semiconductors for optoelectronic applications.