Built in 1769 as a private observatory for King George III, Kew Observatory was taken over in 1842 by the British Association for the Advancement of Science (BAAS). It was then quickly transformed into what some claimed to be a 'physical observatory' of the sort proposed by John Herschel - an observatory that gathered data in a wide range of physical sciences, including geomagnetism and meteorology, rather than just astronomy. Yet this article argues that the institution which emerged in the 1840s was different in many ways from that envisaged by Herschel. It uses a chronological framework to show how, at every stage, the geophysicist and Royal Artillery officer Edward Sabine manipulated the project towards his own agenda: an independent observatory through which he could control the geomagnetic and meteorological research, including the ongoing 'Magnetic Crusade'. The political machinations surrounding Kew Observatory, within the Royal Society and the BAAS, may help to illuminate the complex politics of science in early Victorian Britain, particularly the role of 'scientific servicemen' such as Sabine. Both the diversity of activities at Kew and the complexity of the observatory's origins make its study important in the context of the growing field of the 'observatory sciences'.