In recent years, hydrogel-based three-dimensional tumor models have become increasingly mainstream for cancer research. Hydrogels enable recapitulation of biochemical and biophysical cues in the tumor microenvironment (TME) for the culture of cancer and stromal cells. While there is increasing insight into how cancer-stromal interactions support tumor progression and drug resistance, much remains to be understood for the successful development of therapeutic targets that are capable of controlling tumors in patients. This review aims to first describe both acellular and cellular characteristics of the TME, focusing on cancer cell interactions with the extracellular matrix, fibroblasts, endothelial cells and immune cells. We will then discuss hydrogel systems that have been developed in the past four years to mimic these interactions in the TME and finally propose future directions in the field of in vitro tumor modeling.