Spherocylindrical optical elements can be decomposed into a sphere-equivalent component and two cross-cylinder components, oriented at 45 degrees to one another. These components in turn can be represented with a simple matrix formalism. This matrix formalism allows it to be seen that the components also form members of an eight element group, designated the refractive group. The structure of this group is developed including its algebra and its representation with Cayley diagrams. The group is identified as the eight element dihedral group, D4, and is compared to another well-known eight element group, the quaternion group. An example is given using the group formal algebra to develop the transfer equations for spherocylindrical wavefronts. Certain properties of propagating spherocylindrical wavefronts, such as nonrotation of cylinder axes, are seen to come directly as consequences of the group properties.