Estrogen receptor beta (ERβ) selective ligands have attracted much attention recently in the design of anti-cancer drugs that are devoid of the common side effects of estrogen. Structural studies of estrogen receptor alpha (ERα) and β revealed that there were considerable differences in their ligand-binding cavity and in their volume. Hence, the present study has hypothesized that size and shape descriptors can influence the affinity/selectivity of the ligands towards ERβ. To prove the same, quantitative structure-activity relationship (QSAR) analyses were carried out using multiple regression analysis on 2-phenylquinoline, tetrahydrofluorenone and 3-hydroxy-6H-benzo[c]chromen-6-one series. Results indicate that increased lipophilicity, decrease in ellipsoidal volume and width of substituents, presence of halogen atoms was essential for the ligands to have high affinity/selectivity towards ERβ. QSAR models obtained were both internally and externally validated. The study delineates that the size and shape descriptors are best modulators of ERβ affinity/selectivity. Docking studies were performed to support our QSAR results.