The interplay of bulk and boundary scattering is explored in a regime where quantum size effects modify mesoscopic transport in a degenerate Fermi liquid film of 3He on a rough surface. We discuss mass transport and the momentum relaxation time of the film in a torsional oscillator geometry within the framework of a quasiclassical theory that includes the experimentally determined power spectrum of the rough surface. The theory explains the anomalous temperature dependence of the relaxation rate observed experimentally. We model further studies on 3He confined in nanofluidic sample chambers with lithographically defined surface roughness. The improved understanding of surface roughness scattering can be extended to the analogous system of electrons in metals and suggests routes to improve the conductivity of thin metallic films. © 2012 Springer Science+Business Media New York.