An optically-enabled radio frquency (RF) self-interference cancellation system is demonstrated for over-the-air in-band full duplex transmission, based on a signal-of-interest (SOI) driven regular triangle algorithm. Since the goal of a self-interference cancellation system is to retrieve the SOI that is masked by the in-band interference signal, using the SOI quality as the driven parameter for optimizing the self-interference cancellation performance is a natural and effective way to allow the system to adapt to changes and obtain the best cancellation performance. Since regular triangle algorithm has short iteration time, bursts of pseudo-random binary sequence would be used between real data transmission for optimizing the self-interference cancellation performance. The adaptive regular triangle algorithm optimizes the cancellation setting such that the in-band interference can be cancelled to a minimum, i.e., down to the noise floor. During the over-the-air experiment, 22 dB of cancellation depth is obtained over a 300 MHz bandwidth at 18.35 GHz without the need of digital self-interference cancellation.