The hypothesis that colors could be memorized either in verbal or visual working memory depending on the color category borders was tested using functional magnetic resonance imaging. We introduced a 2-back task to investigate the involvement of verbal and visual working memory in color memory. Colors across the categories, defined by basic color names, strongly activated the left inferior frontal gyrus and left inferior parietal lobule corresponding to the phonological loop as verbal working memory, whereas colors within the same category strongly activated the right inferior frontal gyrus corresponding to the visuospatial sketchpad as visual working memory. The choice of colors to memorize might modulate the cognitive load balance between the phonological loop and the visuospatial sketchpad.