Precision oncology lacks scalable tools to assess, at the patient level, systems-level tumor microenvironment (TME) programs driving therapeutic resistance. To address this gap, we trained a weakly-supervised deep learning model, using routine H&E slides as input, to derive quantitative activity for therapeutically-relevant TME phenotypes, spanning immune, metabolic, and tumor cell-intrinsic programs. Using 3111 breast cancer H&E WSIs with matched bulk transcriptomics, our model accurately infers these biological states (AUROC>0.80; PCC>0.64). Validation spanned three levels: (i) tissue-matched multiplexed immunofluorescence, showing concordance between inferred functional states and immune cell fractions (p=0.006-0.106), (ii) blinded reader assessments, confirming localization of phenotype-specific morphology (p<3×10-5), and (iii) multi-institutional patient cohorts, where model-derived phenotypes stratified for clinical response (p<0.045). Despite relying on slide-level labels for training, our model's attention mechanism identifies focal regions of tumor tissue that drive the overarching clinical phenotype or treatment response. By extracting spatially resolved TME biology from routine histology, this strategy can be applied to massive legacy biobanks to enable discovery of new morpho-molecular mediators of resistance across real-world patient populations.