Graphene is a very attractive material for broadband photodetection in hyperspectral imaging and sensing systems. However, its potential use has been hindered by tradeoffs between the responsivity, bandwidth, and operation speed of existing graphene photodetectors. Here, we present engineered photoconductive nanostructures based on gold-patched graphene nano-stripes, which enable simultaneous broadband and ultrafast photodetection with high responsivity. These nanostructures merge the advantages of broadband optical absorption, ultrafast photocarrier transport, and carrier multiplication within graphene nano-stripes with the ultrafast transport of photocarriers to gold patches before recombination. Through this approach, high-responsivity operation is realized without the use of bandwidth-limiting and speed-limiting quantum dots, defect states, or tunneling barriers. We demonstrate high-responsivity photodetection from the visible to infrared regime (0.6 A/W at 0.8 μm and 11.5 A/W at 20 μm), with operation speeds exceeding 50 GHz. Our results demonstrate improvement of the response times by more than seven orders of magnitude and an increase in bandwidths of one order of magnitude compared to those of higher-responsivity graphene photodetectors based on quantum dots and tunneling barriers.