Bicritical points, at which two distinct symmetry-broken phases become simultaneously unstable, are typical for spin-flop metamagnetism. Interestingly, the heavy-fermion compound YbAgGe also possesses such a bicritical point (BCP) with a low temperature T(BCP)≈0.3 K at a magnetic field of μH(BCP)≈4.5 T. In its vicinity, YbAgGe exhibits anomalous behavior that we attribute to the influence of a quantum bicritical point that is close in parameter space yet can be reached by tuning T(BCP) further to zero. Using high-resolution measurements of the magnetocaloric effect, we demonstrate that the magnetic Grüneisen parameter ΓH indeed both changes sign and diverges as required for quantum criticality. Moreover, ΓH displays a characteristic scaling behavior but only on the low-field side H≲H(BCP), indicating a pronounced asymmetry with respect to the critical field. We speculate that the small value of T(BCP) is related to the geometric frustration of the Kondo lattice of YbAgGe.