We provide analytical arguments showing that the "nonperturbative" approximation scheme to Wilson's renormalization group known as the derivative expansion has a finite radius of convergence. We also provide guidelines for choosing the regulator function at the heart of the procedure and propose empirical rules for selecting an optimal one, without prior knowledge of the problem at stake. Using the Ising model in three dimensions as a testing ground and the derivative expansion at order six, we find fast convergence of critical exponents to their exact values, irrespective of the well-behaved regulator used, in full agreement with our general arguments. We hope these findings will put an end to disputes regarding this type of nonperturbative methods.