Conventional seismic rehabilitation methods may not be suitable for some buildings owing to their high cost and time-consuming foundation work. In recent years, viscoelastic dampers (VEDs) have been widely used in many mid- and high-rise buildings. This study introduces a viscoelastic passive control system called rotary rubber braced damper (RRBD). The RRBD is an economical, lightweight, and easy-to-assemble device. A finite element model considering nonlinearity, large deformation, and material damage is developed to conduct a parametric study on different damper sizes under pushover cyclic loading. The fundamental characteristics of this VED system are clarified by analyzing building structures under cyclic loading. The result show excellent energy absorption and stable hysteresis loops in all specimens. Additionally, by using a sinusoidal shaking table test, the effectiveness of the RRBD to manage the response displacement and acceleration of steel frames is considered. The RRBD functioned at early stages of lateral displacement, indicating that the system is effective for all levels of vibration. Moreover, the proposed damper shows significantly better performance in terms of the column compression force resulting from the brace action compared to chevron bracing (CB).