We address the nature of spin-orbit torques at the magnetic surfaces of topological insulators using the linear-response theory. We find that the so-called Dirac torques in such systems possess a different symmetry compared to their Rashba counterpart, as well as a high anisotropy as a function of the magnetization direction. In particular, the damping torque vanishes when the magnetization lies in the plane of the topological-insulator surface. We also show that the Onsager reciprocal of the spin-orbit torque, the charge pumping, induces an enhanced anisotropic damping. Via a macrospin model, we numerically demonstrate that these features have important consequences in terms of magnetization switching. © 2017 American Physical Society.