We present in detail a Langevin formalism for constructing stochastic dynamical equations for active-matter systems coupled to a thermal bath. We apply the formalism to clarify issues of principle regarding the sources and signatures of nonequilibrium behaviour in a variety of polar and apolar single-particle systems and polar flocks. We show that distance from thermal equilibrium depends on how time-reversal is implemented and hence on the reference equilibrium state. We predict characteristic forms for the frequency-resolved entropy production for an active polar particle in a harmonic potential, which should be testable in experiments.