We aimed to dissociate two levels of hierarchical structure building in language and mathematics, namely 'first-level' (the build-up of hierarchical structure with externally given elements) and 'second-level' (the build-up of hierarchical structure with internally represented elements produced by first-level processes). Using functional magnetic resonance imaging, we investigated these processes in three domains: sentence comprehension, arithmetic calculation (using Reverse Polish notation, which gives two operands followed by an operator) and a working memory control task. All tasks required the build-up of hierarchical structures at the first- and second-level, resulting in a similar computational hierarchy across language and mathematics, as well as in a working memory control task. Using a novel method that estimates the difference in the integration cost for conditions of different trial durations, we found an anterior-to-posterior functional organization in the prefrontal cortex, according to the level of hierarchy. Common to all domains, the ventral premotor cortex (PMv) supports first-level hierarchy building, while the dorsal pars opercularis (POd) subserves second-level hierarchy building, with lower activation for language compared with the other two tasks. These results suggest that the POd and the PMv support domain-general mechanisms for hierarchical structure building, with the POd being uniquely efficient for language.