Although potentially inappropriate medication (PIM) is associated with risk of harm due to adverse effects, it is frequently prescribed for elderly patients. The aim of this qualitative multi-center study was to gain insight into contextual factors that might lead to chronic PIM use. We conducted semi-structured interviews with elderly patients with or without chronic PIM use (patient interviews: n = 52). Patients were between 86 and 96 years old. The participants were recruited from the AgeCoDe study. Interviews were audiotaped and transcribed verbatim. The transcripts of the interviews were analysed using qualitative content analysis. Deductive and inductive categories were determined. We found contextual factors related to the patient and related to patient-general practitioner (GP) communication that might lead to chronic PIM use (i.e., positive features of PIM, maintaining characteristics of medication intake, barriers to deprescribe PIM, external actors supporting PIM intake, system-related factors). Besides certain health-related behaviours (e.g., own obligation to report to GP) and medication-related attitudes and knowledge (e.g., awareness of side effects and interaction of medicines), patient-GP-interactions that were characterised by mutual agreements on drugs (e.g., concerning dosage or discontinuation of a drug) might be advantageous to reduce the probability of chronic PIM use. The results might assist in the development of guidelines and educational programs aiming to reduce PIM use in the elderly.