The extraction of the cardiac Purkinje system (PS) from intensity images is a critical step toward the development of realistic structural models of the heart. Such models are important for uncovering the mechanisms of cardiac disease and improving its treatment and prevention. Unfortunately, the manual extraction of the PS is a challenging and error-prone task due to the presence of image noise and numerous fiber junctions. To deal with these challenges, we propose a framework that estimates local fiber orientations with high accuracy and reconstructs the fibers via tracking. Our key contribution is the development of a descriptor for estimating the orientation distribution function (ODF), a spherical function encoding the local geometry of the fibers at a point of interest. The fiber/branch orientations are identified as the modes of the ODFs via spherical clustering and guide the extraction of the fiber centerlines. Experiments on synthetic data evaluate the sensitivity of our approach to image noise, width of the fiber, and choice of the mode detection strategy, and show its superior performance compared to those of the existing descriptors. Experiments on the free-running PS in an MR image also demonstrate the accuracy of our method in reconstructing such sparse fibrous structures.