Surface roughness becomes relevant if typical length scales of the system are comparable to the variations as it is the case in microfluidic setups. Here, an apparent slip is often detected which can have its origin in the misleading assumption of perfectly smooth boundaries. We investigate the problem by means of lattice Boltzmann simulations and introduce an "effective no-slip plane" at an intermediate position between peaks and valleys of the surface. Our simulations agree with analytical results for sinusoidal boundaries, but can be extended to arbitrary geometries and experimentally obtained data. We find that the apparent slip is independent of the detailed boundary shape, but only given by the distribution of surface heights. Further, we show that slip diverges as the amplitude of the roughness increases which highlights the importance of a proper treatment of surface variations in very confined geometries.