The incorporation of accurately aligned anatomical information as a prior to guide reconstruction and noise regularization in positron emission tomography (PET) has been suggested in many previous studies. However, the advantages of this approach can only be realized if the exact lesion outline is also available. In practice, the anatomical imaging modality may be unable to differentiate between normal and pathological tissues, and thus the edges of lesions seen in the anatomical image may not correspond to functional boundaries in the emission image. In this study, we explored an alternative approach to incorporating an anatomical prior into PET image reconstruction. Of particular interest was the realistic situation where lesions are apparent in the emission images but not in the corresponding anatomical images. In the proposed method, regional information obtained from the anatomical prior was used to estimate an anatomically adaptive anisotropic median-diffusion filtering (AAMDF) prior. This smoothing prior was determined and applied adaptively to each anatomical region on the emission image and then assembled to form a prior image for the next iteration in the reconstruction process. We formulated a two-step joint estimation reconstruction scheme to update the estimated image and prior image iteratively. The proposed AAMDF prior was evaluated and compared with maximum a posteriori (MAP) reconstruction methods with and without anatomical side information. In experiments using synthetic and physical phantom data, the AAMDF prior yielded overall higher lesion-to-background contrast and less error in lesion estimation than other algorithms for a comparable level of background noise. We conclude that lesion contrast and quantification can be improved using an anatomically derived smoothing prior without requiring knowledge of the lesion boundary. This may have important implications in clinical PET/CT, where lesion boundaries are often not obtainable from CT images.