Histone deacetylases (HDACs) are key regulators of gene expression and thereby compelling targets in the treatment of various cancers. Class- and isoform-selective HDAC inhibitors targeting the particular isoform to treat cancers without affecting the normal expression of other isoforms are highly desirable. Molecular dynamics simulations were performed with the set of selective inhibitors and HDAC isoforms of three different classes. The results were compared both within and across the isoforms. The hydrogen bonds between protein and inhibitors are directly correlated with the selective experimental activity. The calculated distances between important amino acids and the metal binding part of inhibitors have disclosed the optimal distance to be maintained by a selective inhibitor. In addition, the calculated non-bonded interaction energies between inhibitor and catalytic residues revealed that the subtle difference in the amino acids at the highly conserved active sites of HDAC isoforms effectively scripts the selectivity story observed experimentally. The results of this study provide valuable information in designing highly selective HDAC inhibitors.