Spatial reasoning has a relevant role in mathematics and helps daily computational activities. It is widely assumed that in cultures with left-to-right reading, numbers are organized along the mental equivalent of a ruler, the mental number line, with small magnitudes located to the left of larger ones. Patients with right brain damage can disregard smaller numbers while mentally setting the midpoint of number intervals. This has been interpreted as a sign of spatial neglect for numbers on the left side of the mental number line and taken as a strong argument for the intrinsic left-to-right organization of the mental number line. Here, we put forward the understanding of this cognitive disability by discovering that patients with right brain damage disregard smaller numbers both when these are mapped on the left side of the mental number line and on the right side of an imagined clock face. This shows that the right hemisphere supports the representation of small numerical magnitudes independently from their mapping on the left or the right side of a spatial-mental layout. In addition, the study of the anatomical correlates through voxel-based lesion-symptom mapping and the mapping of lesion peaks on the diffusion tensor imaging-based reconstruction of white matter pathways showed that the rightward bias in the imagined clock-face was correlated with lesions of high-level middle temporal visual areas that code stimuli in object-centred spatial coordinates, i.e. stimuli that, like a clock face, have an inherent left and right side. In contrast, bias towards higher numbers on the mental number line was linked to white matter damage in the frontal component of the parietal-frontal number network. These anatomical findings show that the human brain does not represent the mental number line as an object with an inherent left and right side. We conclude that the bias towards higher numbers in the mental bisection of number intervals does not depend on left side spatial, imagery or object-centred neglect and that it rather depends on disruption of an abstract non-spatial representation of small numerical magnitudes.