Access to scanners for magnetic resonance imaging (MRI) is typically limited by cost and by infrastructure requirements. Here, we report the design and testing of a portable prototype scanner for brain MRI that uses a compact and lightweight permanent rare-earth magnet with a built-in readout field gradient. The 122-kg low-field (80 mT) magnet has a Halbach cylinder design that results in a minimal stray field and requires neither cryogenics nor external power. The built-in magnetic field gradient reduces the reliance on high-power gradient drivers, lowering the overall requirements for power and cooling, and reducing acoustic noise. Imperfections in the encoding fields are mitigated with a generalized iterative image reconstruction technique that leverages previous characterization of the field patterns. In healthy adult volunteers, the scanner can generate T1-weighted, T2-weighted and proton density-weighted brain images with a spatial resolution of 2.2 × 1.3 × 6.8 mm3. Future versions of the scanner could improve the accessibility of brain MRI at the point of care, particularly for critically ill patients.