Two level systems that can be reliably controlled and measured hold promise as qubits both for metrology and for quantum information science. Since a fluctuating environment limits the performance of qubits in both capacities, understanding environmental coupling and dynamics is key to improving qubit performance. We show measurements of the level splitting and dephasing due to the voltage noise of a GaAs singlet-triplet qubit during exchange oscillations. Unexpectedly, the voltage fluctuations are non-Markovian even at high frequencies and exhibit a strong temperature dependence. This finding has impacts beyond singlet-triplet qubits since nearly all solid state qubits suffer from some kind of charge noise. The magnitude of the fluctuations allows the qubit to be used as a charge sensor with a sensitivity of 2 × 10(-8)e/sqrt[Hz], 2 orders of magnitude better than a quantum-limited rf single electron transistor. Based on these measurements, we provide recommendations for improving qubit coherence, allowing for higher fidelity operations and improved charge sensitivity.