While the nonradiative decay of surface plasmons was once thought to be only a parasitic process that limits the performance of plasmonic devices, it has recently been shown that it can be harnessed in the form of hot electrons for use in photocatalysis, photovoltaics, and photodetectors. Unfortunately, the quantum efficiency of hot electron devices remains low due to poor electron injection and in some cases low optical absorption. Here, we demonstrate how metamaterial perfect absorbers can be used to achieve near-unity optical absorption using ultrathin plasmonic nanostructures with thicknesses of 15 nm, smaller than the hot electron diffusion length. By integrating the metamaterial with a silicon substrate, we experimentally demonstrate a broadband and omnidirectional hot electron photodetector with a photoresponsivity that is among the highest yet reported. We also show how the spectral bandwidth and polarization-sensitivity can be manipulated through engineering the geometry of the metamaterial unit cell. These perfect absorber photodetectors could open a pathway for enhancing hot electron based photovoltaic, sensing, and photocatalysis systems.