The electron-phonon (e-ph) interaction in lead halide perovskites (LHPs) plays a role in a variety of physical phenomena. Unveiling how the local lattice distortion responds to charge carriers is a critical step toward understanding the e-ph interaction in LHPs. Herein, we advance a fundamental understanding of the e-ph interaction in LHPs from the perspective of stereochemical activity of 6s2 lone-pair electrons on the Pb2+ cation. We demonstrate a model system based on three LHPs with distinctive lone-pair activities for studying the structure-property relationships. By tuning the A-cation chemistry, we synthesized single-crystal CsPbBr3, (MA0.13EA0.87)PbBr3 (MA+ = methylammonium; EA+ = ethylammonium), and (MHy)PbBr3 (MHy+ = methylhydrazinium), which exhibit stereo-inactive, dynamic stereo-active, and static stereo-active lone pairs, respectively. This gives rise to distinctive local lattice distortions and low-frequency vibrational modes. We find that the e-ph interaction leads to a blue shift of the band gap as temperature increases in the structure with the dynamic stereo-active lone pair but to a red shift in the structure with the static stereo-active lone pair. Furthermore, analyses of the temperature-dependent low-energy photoluminescence tails reveal that the strength of the e-ph interaction increases with increasing lone-pair activity, leading to a transition from a large polaron to a small polaron, which has significant influence on the emission spectra and charge carrier dynamics. Our results highlight the role of the lone-pair activity in controlling the band gap, phonon, and polaronic effect in LHPs and provide guidelines for optimizing the optoelectronic properties, especially for tin-based and germanium-based halide perovskites, where stereo-active lone pairs are more prominent than their lead counterparts.