Phase retrieval is indispensable for a number of coherent imaging systems. Owing to limited exposure, it is a challenge for traditional phase retrieval algorithms to reconstruct fine details in the presence of noise. In this Letter, we report an iterative framework for noise-robust phase retrieval with high fidelity. In the framework, we investigate nonlocal structural sparsity in the complex domain by low-rank regularization, which effectively suppresses artifacts caused by measurement noise. The joint optimization of sparsity regularization and data fidelity with forward models enables satisfying detail recovery. To further improve computational efficiency, we develop an adaptive iteration strategy that automatically adjusts matching frequency. The effectiveness of the reported technique has been validated for coherent diffraction imaging and Fourier ptychography, with ≈7 dB higher peak SNR (PSNR) on average, compared with conventional alternating projection reconstruction.