Head-mounted display (HMD) has been widely used in many fields, and most existing HMDs are complex and typically non-aesthetically pleasing. In this paper, a novel compact, lightweight and wearable off-axis HMD with freeform surface is reported. It is challenging to achieve large field of view (FOV) and maintain compact structure simultaneously for this type system. In this design, the compact off-axis HMD consists of a tilted ellipsoid combiner and a four pieces relay lenses. It offers a diagonal FOV of 30°, and an exit pupil diameter of 7 mm. No diffractive surfaces are used, thus avoiding the effect of stray light and ghost image in previous designs. The x-y polynomial freeform surface is adopted in the relay lens for improving the image quality and minimizing the structure size. Analytical expressions to determine the initial structure of HMD has been given, while structure constrains, optimization strategy and tolerance analysis are also described in details. The optical system is easy to manufacture by ordinary method which is beneficial for mass production. Further, a prototype of this compact HMD is successfully implemented with good imaging performance. The compact structure of HMD makes it well suited for integrating a normal glass, significantly advancing the application of HMD in people's daily life.