The activity of the avian myeloblastosis virus (AMV) or the human immunodeficiency virus type 1 (HIV-1) protease on peptide substrates which represent cleavage sites found in the gag and gag-pol polyproteins of Rous sarcoma virus (RSV) and HIV-1 has been analyzed. Each protease efficiently processed cleavage site substrates found in their cognate polyprotein precursors. Additionally, in some instances heterologous activity was detected. The catalytic efficiency of the RSV protease on cognate substrates varied by as much as 30-fold. The least efficiently processed substrate, p2-p10, represents the cleavage site between the RSV p2 and p10 proteins. This peptide was inhibitory to the AMV as well as the HIV-1 and HIV-2 protease cleavage of other substrate peptides with Ki values in the 5-20 microM range. Molecular modeling of the RSV protease with the p2-p10 peptide docked in the substrate binding pocket and analysis of a series of single-amino acid-substituted p2-p10 peptide analogues suggested that this peptide is inhibitory because of the potential of a serine residue in the P1' position to interact with one of the catalytic aspartic acid residues. To open the binding pocket and allow rotational freedom for the serine in P1', there is a further requirement for either a glycine or a polar residue in P2' and/or a large amino acid residue in P3'. The amino acid residues in P1-P4 provide interactions for tight binding of the peptide in the substrate binding pocket.