The low-temperature dynamics of quantum systems are dominated by the low-energy eigenstates. For two-dimensional systems in particular, exotic phenomena such as topological order and anyon excitations can emerge. While a complete low-energy description of strongly correlated systems is hard to obtain, essential information about the elementary excitations is encoded in the eigenvalue structure of the quantum transfer matrix. Here we study the transfer matrix of topological quantum systems using the tensor network formalism and demonstrate that topological quantum order requires a particular type of 'symmetry breaking' for the fixed point subspace. We also relate physical anyon excitations to domain-wall excitations at the level of the transfer matrix. This formalism enables us to determine the structure of the topological sectors in two-dimensional gapped phases very efficiently, therefore opening novel avenues for studying fundamental questions related to anyon condensation and confinement.