A network is frustrated when competing interactions between nodes prevent each bond from being satisfied. This compromise is central to the behaviour of many complex systems, from social and neural networks to protein folding and magnetism. Frustrated networks have highly degenerate ground states, with excess entropy and disorder even at zero temperature. In the case of quantum networks, frustration can lead to massively entangled ground states, underpinning exotic materials such as quantum spin liquids and spin glasses. Here we realize a quantum simulation of frustrated Ising spins in a system of three trapped atomic ions, whose interactions are precisely controlled using optical forces. We study the ground state of this system as it adiabatically evolves from a transverse polarized state, and observe that frustration induces extra degeneracy. We also measure the entanglement in the system, finding a link between frustration and ground-state entanglement. This experimental system can be scaled to simulate larger numbers of spins, the ground states of which (for frustrated interactions) cannot be simulated on a classical computer.