In this work, we have prepared a series of polydimethylsiloxane (PDMS) composites containing various graphene flakes loadings (0.02-2 wt%), and their broadband optical properties are being investigated. We demonstrate the tunability and evolution of transmittance and reflection spectra of the composites in a wide spectral range (0.4-200 μm) as a function of graphene content. Using these data we derive the broadband wavelength-dependent absorption coefficient (α) values. Our results show that α is roughly constant in the visible and IR ranges, and, surprisingly, is approximately one order of magnitude lower in the terahertz regime, suggesting different terahertz radiation scattering mechanism in our composite. Our material could be useful for applications in optical communication, sensing or ultrafast photonics.