Recently, we have developed a multilayer energy-based fragment (MLEBF) method to describe excited states of large systems in which photochemically active and inert regions are separately treated with multiconfigurational and single-reference electronic structure method and their mutual polarization effects are naturally described within the many-body expansion framework. This MLEBF method has been demonstrated to provide highly accurate energies and gradients. In this work, we have further derived the MLEBF method with which highly accurate excited-state Hessian matrices of large systems are efficiently constructed. Moreover, in combination with recently proposed embedded atom neural network (EANN) model we have developed a machine learning (ML) accelerated MLEBF method (i.e., ML-MLEBF) in which photochemically inert region is entirely replaced with trained ML models. ML-MLEBF is found to improve computational efficiency of Hessian matrices in particular for large systems. Furthermore, both MLEBF and ML-MLEBF methods are highly parallel and exhibit low-scaling computational cost with multiple CPUs. The present developments could motivate combining various ML techniques with fragment-based electronic structure methods to explore Hessian-matrix-based excited-state properties of large systems.