This article studies the detection of discontinuous false data-injection (FDI) attacks on cyber-physical systems (CPSs). Considering the unknown stochastic properties of the process noise and measurement noise, deep reinforcement learning is applied to designing an FDI attack detector. First, the discontinuous attack detection problem is modeled as a partially observable Markov decision process (POMDP) and a neural network is used to explore the POMDP. In the network, sliding observation windows which are composed of the offline fragment historical data are used as the input. An approach to designing the reward in POMDP is provided to ensure the precision of the detection when there are even some state recognition errors. Second, sufficient conditions on attack frequency and duration to guarantee the applicability of the detector and the expected estimation performance are further given. Finally, simulation examples illustrate the effectiveness of the attack detector.