A zeroth-order Hamiltonian based on Koopmans matrices for complete active space second-order perturbation theory (CASPT2) is presented. This Hamiltonian involves three types of Fock matrices. The original CASPT2 Fock matrix is retained for excitation classes where the excitation does not change the number of electrons in the complete active space (CAS). For excitation classes involving a change in the number of electrons in the CAS, two alternative Fock matrices corresponding to either positive or negative ionization of the CAS are introduced. These are constructed such that they exactly reproduce the Koopmans matrices for a singly ionized CAS. Test calculations indicate that the method gives better excitation energies than CASPT2 without using empirical parameters, for example, the ionization potential-electron affinity shift, which is also designed to improve excitation energies. The method is also less prone to intruder states than conventional CASPT2. Moreover, the dissociation curve for the chromium dimer looks much more reasonable than the one obtained with conventional CASPT2.