The observation of a metallic interface between band insulators LaAlO$_3$ and SrTiO$_3$ has led to massive efforts to understand the origin of the phenomenon as well as to search for other systems hosting such two-dimensional electron gases (2-DEGs). However, the understanding of the origin of the 2-DEG is very often hindered as several possible mechanisms such as polar catastrophe, cationic intermixing, and oxygen vacancy (OV) can be operative simultaneously. The presence of a heavy element makes the KTaO$_3$ (KTO) based 2-DEG a potential platform to investigate spin-orbit coupling driven novel electronic and magnetic phenomena. In this work, we investigate the sole effect of OV in KTO, which makes KTO metallic. O K-edge x-ray absorption spectroscopy measurements find that OV dopes electrons in Ta t$^*_{2g}$ antibonding states. Photoluminescence measurements reveal the existence of a highly localized deep midgap state in oxygen-deficient KTO. Our detailed ab initio calculations demonstrate that such deep midgap state arises due to the linear clustering of OVs around Ta. Our present work emphasizes that we must pay attention to the possible presence of OVs in interpreting emergent behavior of KTO-based heterostructures.