We present an open-geometry Fourier modal method based on a new combination of open boundary conditions and an efficient k-space discretization. The open boundary of the computational domain is obtained using basis functions that expand the whole space, and the integrals subsequently appearing due to the continuous nature of the radiation modes are handled using a discretization based on nonuniform sampling of the k space. We apply the method to a variety of photonic structures and demonstrate that our method leads to significantly improved convergence with respect to the number of degrees of freedom, which may pave the way for more accurate and efficient modeling of open nanophotonic structures.