We present promising results of novel high-temperature superconducting (HTS) shim coil prototypes that circumvent the size and strength limitation of our earlier innovative HTS shim concept based on 46-mm wide REBCO tape. The HTS shim coil is placed inside the HTS magnet, mainly for ultra-high-field (> 1 GHz or 23.5 T) NMR magnets, and thus unaffected from the windings' diamagnetic wall effects. One full-scale version will be applied to clean up Z1 and Z2 harmonic errors in the MIT 1.3-GHz high-resolution NMR magnet composed of an 835-MHz HTS insert, while another version for an MIT 1-GHz microcoil NMR magnet whose small-scale model we are currently building. The prototype sets were wound with a 2-pile, 1.03-mm wide, 0.30-mm thick REBCO conductor. Operated at 77 K, the Z1 shim set generated a 1st harmonic field strength of 179 kHz/cm at 70 A, while the Z2 shim set, composed of two pairs, Z21 and Z22, generated the 2nd harmonic field of 141 kHz/cm2 at 50 A. Together with discussion on technical challenges for this REBCO shim coil concept, we demonstrate its feasibility for the next generation of ultra-high-field (UHF) HTS NMR magnets.