Hearing impaired and normal hearing individuals were compared in two within-participant office noise conditions (high noise: 60 L Aeq and low noise: 30 L Aeq ). Performance, subjective fatigue, and physiological stress were tested during working on a simulated open-plan office. We also tested two between-participants restoration conditions following the work period with high noise (nature movie or continued office noise). Participants with a hearing impairment (N = 20) were matched with normal hearing participants (N = 18) and undertook one practice session and two counterbalanced experimental sessions. In each experimental session they worked for two hours with basic memory and attention tasks. We also measured physiological stress indicators (cortisol and catecholamines) and self-reports of mood and fatigue. The hearing impaired participants were more affected by high noise than the normal hearing participants, as shown by impaired performance for tasks that involve recall of semantic information. The hearing impaired participants were also more fatigued by high noise exposure than participants with normal hearing, and they tended to have higher stress hormone levels during the high noise compared to the low noise condition. Restoration with a movie increased performance and motivation for the normal hearing participants, while rest with continued noise did not. For the hearing impaired participants, continued noise during rest increased motivation and performance, while the movie did not. In summary, the impact of noise and restorative conditions varied with the hearing characteristics of the participants. The small sample size does however encourage caution when interpreting the results.