Spin-orbit torques (SOTs) have been widely understood as an interfacial transfer of spin that is independent of the bulk properties of the magnetic layer. Here, we report that SOTs acting on ferrimagnetic FexTb1-x layers decrease and vanish upon approaching the magnetic compensation point because the rate of spin transfer to the magnetization becomes much slower than the rate of spin relaxation into the crystal lattice due to spin-orbit scattering. These results indicate that the relative rates of competing spin relaxation processes within magnetic layers play a critical role in determining the strength of SOTs, which provides a unified understanding for the diverse and even seemingly puzzling SOT phenomena in ferromagnetic and compensated systems. Our work indicates that spin-orbit scattering within the magnet should be minimized for efficient SOT devices. We also find that the interfacial spin-mixing conductance of interfaces of ferrimagnetic alloys (such as FexTb1-x) is as large as that of 3d ferromagnets and insensitive to the degree of magnetic compensation. © 2023, The Author(s).