The transport of intensity equation (TIE) is a two-dimensional second order elliptic partial differential equation that must be solved under appropriate boundary conditions. However, the boundary conditions are difficult to obtain in practice. The fast Fourier transform (FFT) based TIE solutions are widely adopted for its speed and simplicity. However, it implies periodic boundary conditions, which lead to significant boundary artifacts when the imposed assumption is violated. In this work, TIE phase retrieval is considered as an inhomogeneous Neumann boundary value problem with the boundary values experimentally measurable around a hard-edged aperture, without any assumption or prior knowledge about the test object and the setup. The analytic integral solution via Green's function is given, as well as a fast numerical implementation for a rectangular region using the discrete cosine transform. This approach is applicable for the case of non-uniform intensity distribution with no extra effort to extract the boundary values from the intensity derivative signals. Its efficiency and robustness have been verified by several numerical simulations even when the objects are complex and the intensity measurements are noisy. This method promises to be an effective fast TIE solver for quantitative phase imaging applications.