Kohonen's self-organizing map (SOM) is used to map high-dimensional data into a low-dimensional representation (typically a 2-D or 3-D space) while preserving their topological characteristics. A major reason for its application is to be able to visualize data while preserving their relation in the high-dimensional input data space as much as possible. Here, we are seeking to go further by incorporating semantic meaning in the low-dimensional representation. In a conventional SOM, the semantic context of the data, such as class labels, does not have any influence on the formation of the map. As an abstraction of neural function, the SOM models bottom-up self-organization but not feedback modulation which is also ubiquitous in the brain. In this paper, we demonstrate a hierarchical neural network, which learns a topographical map that also reflects the semantic context of the data. Our method combines unsupervised, bottom-up topographical map formation with top-down supervised learning. We discuss the mathematical properties of the proposed hierarchical neural network and demonstrate its abilities with empirical experiments.