Hydrogels are often employed for tissue engineering and moistening applications. However, they are rarely used for load-bearing purposes because of their limited stiffness and the stiffness-toughness compromise inherent to them. By contrast, nature uses hydrogel-based materials as scaffolds for load-bearing and protecting materials by mineralizing them. Inspired by nature, the stiffness or toughness of synthetic hydrogels has been increased by forming minerals, such as CaCO3, within them. However, the degree of hydrogel reinforcement achieved with CaCO3 remains limited. To address this limitation, we form CaCO3 biominerals in situ within a model hydrogel, poly(acrylamide) (PAM), and systematically investigate the influence of the size, structure, and morphology of the reinforcing CaCO3 on the mechanical properties of the resulting hydrogels. We demonstrate that especially the structure of CaCO3 and its affinity to the hydrogel matrix strongly influence the mechanical properties of mineralized hydrogels. For example, while the fracture energy of PAM hydrogels is increased 3-fold if reinforced with individual micro-sized CaCO3 crystals, it increases by a factor of 13 if reinforced with a percolating amorphous calcium carbonate (ACC) nano-structure that forms in the presence of a sufficient quantity of Mg2+. If PAM is further functionalized with acrylic acid (AA) that possesses a high affinity towards ACC, the stiffness of the hydrogel increases by a factor 50. These fundamental insights on the structure-mechanical property relationship of hydrogels that have been functionalized with in situ formed minerals has the potential to enable tuning the mechanical properties of mineralized hydrogels over a much wider range than what is currently possible.