As an effort to develop therapeutics for cancer treatments, a number of effective histone deacetylase inhibitors with structural diversity have been discovered. To gain insight into optimizing the activity of an identified lead compound, a computational protocol sequentially involving homology modeling, docking experiments, molecular dynamics simulation, and free energy perturbation calculations was applied for rationalizing the relative activities of known histone deacetylase inhibitors. With the newly developed force field parameters for the coordination environment of the catalytic zinc ion in hand, the computational strategy proved to be successful in predicting the rank orders for 12 derivatives of three hydroxamate-based inhibitor scaffolds with indole amide, pyrrole, and sulfonamide moieties. The results showed that the free energy of an inhibitor in aqueous solution should be an important factor in determining the binding free energy. Hence, in order to enhance the inhibitory activity by adding or substituting a chemical group, the increased stabilization in solution due to the structural changes must be overcome by a stronger enzyme-inhibitor interaction. It was also found that to optimize inhibitor potency, the hydrophobic head of an inhibitor should be elongated or enlarged so that it can interact with Pro29 and His28 that are components of the flexible loop at the top of the active site.