Rapid autofocusing over long distances is critical for tracking 3D topological variations and sample motion in real time. Taking advantage of a deformable mirror and Shack-Hartmann wavefront sensor, remote focusing can permit fast axial scanning with simultaneous correction of system-induced aberrations. Here, we report an autofocusing technique that combines remote focusing with sequence-dependent learning via a bidirectional long short term memory network. A 120 µm autofocusing range was achieved in a compact reflectance confocal microscope both in air and in refractive-index-mismatched media, with similar performance under arbitrary-thickness liquid layers up to 1 mm. The technique was validated on sample types not used for network training, as well as for tracking of continuous axial motion. These results demonstrate that the proposed technique is suitable for real-time aberration-free autofocusing over a large axial range, and provides unique advantages for biomedical, holographic and other related applications.