Sensing of leakage at an early stage is crucial for the safe utilization of hydrogen. Optical hydrogen sensors eliminate the potential hazard of ignition caused by electrical sparks but achieve a detection limit far higher than their electrical counterparts so far. To essentially improve the performance of optical hydrogen sensors in terms of detection limit, we demonstrate in this work a plasmonic hydrogen sensor based on aluminum-palladium (Al-Pd) hybrid nanorods. Arranged into high-density regular arrays, the hybrid nanorods are capable of sensing hydrogen at a concentration down to 40 ppm, i.e., one thousandth of the lower flammability limit of hydrogen in air. Different sensing behaviors are found for two sensor configurations, where Pd-Al nanorods provide larger spectral shift and Al-Pd ones exhibit shorter response time. In addition, the plasmonic hydrogen sensors here utilize exclusively CMOS-compatible materials, holding the potential for real-world, large-scale applications.