In this paper, a novel two-stage subspace-based direction of arrival (DOA) estimation algorithm with the nested array is proposed for mixed signals containing circular and non-circular ones. By exploiting the difference between the two types of steering vectors, the DOAs of circular signals are estimated in the first stage. After eliminating the estimated information of circular signals by the covariance matrix reconstruction and oblique projection methods, the dimensions of the noise subspace are increased in estimating the DOAs of non-circular signals in the second stage. Through the two-stage estimation, the DOAs of the circular and non-circular signals are estimated separately and different types of signals with similar or the same DOAs can be distinguished. Furthermore, to avoid the two-dimensional (2-D) search with huge computational burden, a one-dimensional (1-D) search method exploiting the rank deficiency is proposed in the DOA estimation for non-circular signals. The simulation results show that the proposed algorithm can effectively improve the estimation accuracy and resolution probability, especially when circular and non-circular signals have similar DOAs.
Keywords: mixed signals; nested array; oblique projection; sum-difference co-array.