We have examined the specificity of planar carboxyl and tetrahedral phosphonyl esters for mouse cholinesterases and have delineated the orientation of these ligands in the enzyme active center. The approach involved altering acyl pocket dimensions by site-specific mutagenesis of two phenylalanines and varying ligand size and enantiomer presentation. Substrate catalysis rates by wild type acetylcholinesterase (AChE) of acetyl-, butyryl-, and benzoylthiocholine diminished with increasing size of the acyl moiety. In contrast, substitution of the acyl pocket phenylalanines giving the mutants F295L and F297I of AChE yielded more efficient catalysis of the larger substrates and a specificity approaching that of butyrylcholinesterase. Extension from planar substrates to enantiomerically pure organophosphonates allowed for an analysis of enantiomeric selectivity. We found that AChE reactions are 200-fold faster with the Sp than the Rp enantiomer of of cycloheptyl methylphosphonyl thiocholine. Upon the acyl pocket size being enlarged, the Rp enantiomer became more reactive while reaction with the Sp enantiomer was slightly reduced. In fact, the F297I mutant displayed inverted stereospecificity. A visual correlation with the kinetic data has been developed by docking the ligands in the active site. Upon placement of the phosphonyl oxygen in the oxyanion hole and the leaving group being directed out of the gorge, the Rp, but not the Sp, enantiomer engendered steric hindrance between the alkoxyl group and the acyl pocket. Replacing F297 with Ile accommodated the bulky alkoxyl group of the Rp isomer in the acyl pocket, allowing similar orientations of the phosphonyl oxygen and the leaving group to the Sp isomer.(ABSTRACT TRUNCATED AT 250 WORDS)