The development of transverse instability in the radiation-pressure-acceleration dominant laser-foil interaction is numerically examined by two-dimensional particle-in-cell simulations. When a plane laser impinges on a foil with modulated surface, the transverse instability is incited, and periodic perturbations of the proton density develop. The growth rate of the transverse instability is numerically diagnosed. It is found that the linear growth of the transverse instability lasts only a few laser periods, then the instability gets saturated. In order to optimize the modulation wavelength of the target, a method of information entropy is put forward to describe the chaos degree of the transverse instability. With appropriate modulation, the transverse instability shows a low chaos degree, and a quasi-monoenergetic proton beam is produced.