One of the key objectives in spatial transcriptomics (ST) studies is to map the complex organization and functions of tissues. We introduce GraphScrDom, a reference-informed and weakly supervised contrastive learning model that uniquely integrates expert-provided manual annotations (i.e., scribbles) on spatial grids or histology images with cell type-specific gene expression profiles derived from reference single-cell RNA-seq data to perform tissue segmentation. With only limited scribble annotations, GraphScrDom consistently outperforms existing methods across various ST platforms and at both bulk and single-cell resolutions, as evaluated by six widely used metrics, demonstrating strong generalizability and robustness. Additionally, we have developed an integrative software toolkit that includes an interactive annotation interface and a model training module for spatial domain detection, providing a unified and user-friendly framework to facilitate spatial domain analysis.