Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal human cancers, and is known for its extremely poor prognosis. Because of the location and composition of the organ, early symptoms cannot be visualized as easily as in other solid tumors. In the past decades, researchers have been mostly working on the genetic and epigenetic alterations of the cancer cells themselves, and therapies on pancreatic cancer cells alone have failed to significantly improve patient outcome. With the identification of abundant tumor stromal responses, the focus of pancreatic cancer research has begun to change. Increasing evidence has proved that the tumor stroma, especially the cell components (such as pancreatic stellate cells, tumor-associated macrophages, mast cells etc.) plays a key role in the development of PDAC. In this review, we discuss the interactions between cancer cells and several important cell components of the tumor stroma, as well their role in tumor growth, migration, invasion, angiogenesis and immune recognition.