Multi-scale approaches have been widely studied in pathology image analysis. These offer an ability to characterize tissues in an image at various scales, in which the tissues may appear differently. Many of such methods have focused on extracting multi-scale hand-crafted features and applied them to various tasks in pathology image analysis. Even, several deep learning methods explicitly adopt the multi-scale approaches. However, most of these methods simply merge the multi-scale features together or adopt the coarse-to-fine/fine-to-coarse strategy, which uses the features one at a time in a sequential manner. Utilizing the multi-scale features in a cooperative and discriminative fashion, the learning capabilities could be further improved. Herein, we propose a multi-scale approach that can identify and leverage the patterns of the multiple scales within a deep neural network and provide the superior capability of cancer classification. The patterns of the features across multiple scales are encoded as a binary pattern code and further converted to a decimal number, which can be easily embedded in the current framework of the deep neural networks. To evaluate the proposed method, multiple sets of pathology images are employed. Under the various experimental settings, the proposed method is systematically assessed and shows an improved classification performance in comparison to other competing methods.