We prove a theorem that shows the degeneracy of many-body states for particles in a periodic lattice and under a uniform magnetic field depends on the total particle number and the flux filling ratio. Noninteracting fermions and weakly interacting bosons are given as two examples. For the latter case, the phenomenon can also be physically understood in terms of destructive quantum interference of multiple symmetry-related tunneling paths between classical energy minima, which is reminiscent of the spin-parity effect discovered in magnetic molecular clusters. We also show that the quantum ground state of a mesoscopic number of bosons in this system is not a simple mean-field state but a fragmented state even for very weak interactions. © 2010 The American Physical Society.