We experimentally investigate the origin of two-magnon scattering (TMS) in heavy-metal (HM)/ferromagnet (FM)/oxide heterostructures (FM = Co, Ni81Fe19, or Fe60Co20B20) by varying the materials located above and below the FM layer. We show that strong TMS in HM/FM/oxide systems arises primarily at the HM/FM interface and increases with the strength of the interfacial spin-orbit coupling and magnetic roughness at this interface. TMS at the FM/oxide interface is relatively weak, even in systems where spin-orbit coupling at this interface generates strong interfacial magnetic anisotropy. We also suggest that the spin-current-induced excitation of nonuniform short-wavelength magnons at the HM/FM interface may function as a mechanism of spin memory loss for the spin-orbit torque exerted on the uniform mode. © 2020 American Physical Society.