Yellow catfish (Pelteobagrus fulvidraco) is a pivotal freshwater aquaculture species in China. It shows sexual size dimorphism favoring male in growth. Whole transcriptome approach is required to get the overview of genetic toolkit for understanding the sex determination mechanism aiming at devising its monosex production. Beside gonads, the brain is also considered as a major organ for vertebrate reproduction. Transcriptomic analyses on the brain and of different developmental stages will provide the dynamic view necessary for better understanding its sex determination. In this regard, we have performed a de novo assembly of yellow catfish brain transcriptome by high throughput Illumina sequencing. A total number of 154,507 contigs were obtained with the lengths ranging from 201 to 27,822 bp and N50 of 2,101 bp, as well as 20,699 unigenes were identified. Of these unigenes, 13 and 54 unigenes were detected to be XY-specifically expressed genes (SEGs) for one and 2-year-old yellow catfish, while the corresponding numbers of XX-SEGs for those two stages were 19 and 13, respectively. Our work identifies a set of annotated genes that are candidate factors affecting sexual dimorphism as well as simple sequence repeat (SSR) and single nucleotide variation (SNV) in yellow catfish. To validate the expression patterns of the sex-related genes, we performed quantitative real-time PCR (qRT-PCR) indicating the reliability and accuracy of our analysis. The results in our study may enhance our understanding of yellow catfish sex determination and potentially help to improve the production of all-male yellow catfish for aquaculture.