Magnetic systems with frustration often have large classical degeneracy. We show that their low-energy physics can be understood as dynamics within the space of classical ground states. We demonstrate this mapping in a family of quantum spin clusters where every pair of spins is connected by an XY antiferromagnetic bond. The dimer with two spin-S spins provides the simplest example-it maps to a quantum particle on a ring (S-1). The trimer is more complex, equivalent to a particle that lives on two disjoint rings (S-1 circle times Z(2)). It has an additional subtlety for half-integer S values, wherein both rings must be threaded by pi fluxes to obtain a satisfactory mapping. This is a consequence of the geometric phase incurred by spins. For both the dimer and the trimer, the validity of the effective theory can be seen from a path-integral-based derivation. This approach cannot be extended to the quadrumer which has a nonmanifold ground-state space, consisting of three tori that touch pairwise along lines. In order to understand the dynamics of a particle in this space, we develop a tight-binding model with this connectivity. Remarkably, this successfully reproduces the low-energy spectrum of the quadrumer. For half-integer spins, a geometric phase emerges which can be mapped to two pi-flux tubes that reside in the space between the tori. The nonmanifold character of the space leads to a remarkable effect-the dynamics at low energies is not ergodic as the particle is localized around singular lines of the ground-state space. The low-energy spectrum consists of an extensive number of bound states formed around singularities. Physically, this manifests as an order-by-disorder-like preference for collinear ground states. However, unlike order-by-disorder, this ``order by singularity″ persists even in the classical limit. We discuss consequences for field theoretic studies of magnets.