Several recent behavioral studies have shown that the enumeration of a small number of items (a process termed subitizing) depends on the availability of attentional resources and is not a preattentive process as previously thought. Here we studied the neural correlates of visual enumeration under different attentional loads in a dual-task paradigm using fMRI. Relatively intact subitizing under low attentional load compared to impaired subitizing under high attentional load was associated with an increase in BOLD signal in the right temporo-parietal junction (rTPJ). Crucially, attentionally modulated response in the rTPJ was specific to small set sizes (up to 3 items) and did not occur at larger set sizes (5-7 items). This result has two implications: (1) Subitizing involves part of the fronto-parietal network for stimulus-driven attention providing neural evidence against preattentive subitizing. (2) Activity in rTPJ is set-size modulated. Together with similar evidence from studies probing visual short-term memory, this result suggests that rTPJ modulation might reflect the brain's ability to attentively handle small set sizes. Thus, the rTPJ may play an important role for the emergence of a capacity limit in both enumeration and visual short-term memory.