The feasibility of using certain types of binary phase-only filter (BPOF) is investigated. A critical aspect of correlation filters is not often addressed in research on BPOF's: how well do they perform as classifiers in the presence of imperfectly matching templates? It is not enough to detect a single given signal in the presence of noise; it is equally critical to make the correct classification among a number of possible templates with a low false-alarm rate. We show that (+1, -1)-valued BPOF's based on the real part of a conventional matched filter can cause misclassification of simple patterns, even in the absence of noise. These are known to be suboptimal, but the seriousness of their limitations illustrates an important design issue. It is therefore concluded that other types of filters must be used for correlator-based neural network implementations and image processing in general. We also include a commentary on the potential for facing this type of problem with general POF's and BPOF's. The theoretical results are supported by computer simulation and optical experiments.