Rogue waves are unexpectedly large and localized displacements from an equilibrium position or an otherwise calm background. For the nonlinear Schrödinger (NLS) model widely used in fluid mechanics and optics, these waves can occur only when dispersion and nonlinearity are of the same sign, a regime of modulation instability. For coupled NLS equations, rogue waves will arise even if dispersion and nonlinearity are of opposite signs in each component as new regimes of modulation instability will appear in the coupled system. The same phenomenon will be demonstrated here for a coupled "AB" system, a wave-current interaction model describing baroclinic instability processes in geophysical flows. Indeed, the onset of modulation instability correlates precisely with the existence criterion for rogue waves for this system. Transitions from "elevation" rogue waves to "depression" rogue waves are elucidated analytically. The dispersion relation as a polynomial of the fourth order may possess double pairs of complex roots, leading to multiple configurations of rogue waves for a given set of input parameters. For special parameter regimes, the dispersion relation reduces to a cubic polynomial, allowing the existence criterion for rogue waves to be computed explicitly. Numerical tests correlating modulation instability and evolution of rogue waves were conducted.