Oscillatory phenomena in the brain activity and their synchronization are frequently studied using mathematical models and analytic tools derived from nonlinear dynamics. In many experimental situations, however, neural signals have a broadband character and if oscillatory activity is present, its dynamical origin is unknown. To cope with these problems, a framework for detecting nonlinear oscillatory activity in broadband time series is presented. First, a narrow-band oscillatory mode is extracted from a broadband background. Second, it is tested whether the extracted mode is significantly different from linearly filtered noise, modelled as a linear stochastic process possibly passed through a static nonlinear transformation. If a nonlinear oscillatory mode is positively detected, further analysis using nonlinear approaches such as the phase synchronization analysis can potentially bring new information. For linear processes, however, standard approaches such as the coherence analysis are more appropriate and provide sufficient description of underlying interactions with smaller computational effort. The method is illustrated in a numerical example and applied to analyze experimentally obtained human EEG time series from a sleeping subject.