A 3-D ultrasound (US) imaging technique has been studied to facilitate the diagnosis of spinal deformity without radiation. The objective of this article is to propose an assessment framework to automatically estimate spinal deformity in US spine images. The proposed framework comprises four major components, a US spine image generator, a novel transformer-based lightweight spine detector network, an angle evaluator, and a 3-D modeler. The principal component analysis (PCA) and discriminative scale space tracking (DSST) method are first adopted to generate the US spine images. The proposed detector is equipped with a redundancy queries removal (RQR) module and a regularization item to realize accurate and unique detection of spine images. Two clinical datasets, a total of 273 images from adolescents with idiopathic scoliosis, are used for the investigation of the proposed framework. The curvature is estimated by the angle evaluator, and the 3-D mesh model is established by the parametric modeling technique. The accuracy rate (AR) of the proposed detector can be achieved at 99.5%, with a minimal redundancy rate (RR) of 1.5%. The correlations between automatic curve measurements on US spine images from two datasets and manual measurements on radiographs are 0.91 and 0.88, respectively. The mean absolute difference (MAD) and standard deviation (SD) are 2.72° ± 2.14° and 2.91° ± 2.36° , respectively. The results demonstrate the effectiveness of the proposed framework to advance the application of the 3-D US imaging technique in clinical practice for scoliosis mass screening and monitoring.