Neurite growth is a fundamental process of neuronal development, which requires both membrane expansions by exocytosis and cytoskeletal dynamics. However, the specific contribution of these processes has not been yet assessed quantitatively. To study and quantify the growth process, we construct a biophysical model in which we relate the overall neurite outgrowth rate to the vesicle dynamics. By considering the complex motion of vesicles in the cell soma, we demonstrate from biophysical consideration that the main step of finding the neurite initiation site relies mainly on a two-dimensional diffusion/sequestration/fusion at the cell surface and we obtain a novel formula for the flux of vesicles at the neurite base. In the absence of microtubules, we show that a nascent neurite initiated by vesicular delivery can only reach a small length. By adding the microtubule dynamics to the secretory pathway and using stochastic analysis and simulations, we study the complex dynamics of neurite growth. Within this model, depending on the coupling parameter between the microtubules and the neurite, we find different regimes of growth, which describe dendritic and axonal growth. To validate one aspect of our model, we demonstrate that the experimental flux of TI-VAMP but not Synaptobrevin 2 vesicles contributes to the neurite growth. We conclude that although vesicles can be generated randomly in the cell body, the search for the neurite position using the microtubule network and diffusion is quite fast. Furthermore, when the TI-VAMP vesicular flow is large enough, the interactions between the microtubule bundle and the neurite control the growth process. In addition, all of these processes intimately cooperate to mediate the various modes of neurite growth: the model predicts three different growing modes including, in addition to the stable axonal growth and the stochastic dendritic growth, a fast oscillatory regime. Finally our study demonstrates that cytoskeletal dynamics is necessary to generate long protrusion, while vesicular delivery alone can only generate small neurite.