Conserved growth models that exhibit nonlinear instability in which the height ( depth) of isolated pillars (grooves) grows in time are studied by numerical integration and stochastic simulation. When this instability is controlled by the introduction of an infinite series of higher-order nonlinear terms, these models exhibit, as a function of control parameter, a nonequilibrium phase transition between a kinetically rough phase with self-affine scaling and phase that exhibits mound formation, slope selection and power law coarsening.