The lateral prefrontal cortex is known to be organized by cognitive hierarchies following a posterior-to-anterior gradient. Here we test whether this model applies across different cognitive domains by varying levels of cognitive hierarchy in first language, second language and non-language domains. These domains vary in their degree of automaticity with first language being the most automatic. For second language/non-language a clear gradient pattern of activation depending on the level of hierarchy is observed in the prefrontal cortex with the highest level of hierarchy recruiting its most anterior region, whereas for first language the highest level of hierarchy recruits its most posterior region. Moreover, second language/non-language and first language differ in the structural connectivity of their underlying networks. The current data strongly suggest that functional segregation of the prefrontal cortex is determined by cognitive hierarchy and the degree of automaticity.