The Cosmic Microwave Background is one of the strongest probes of the cosmological history of our Universe. Several experiments from ground as well as from space continue to perform measurements of the CMB with increasing sensitivity. As the detected signal contains bright microwave emission from our Galaxy as well, the careful separation of the foreground from the background becomes a delicate act. Approaching the subject from di↵erent perspectives, I argue that the physical mechanisms of foregrounds must be better understood before attempting the most ambitious measurements of primordial physics.Residual emission from Galactic Radio Loop I in current CMB maps occupies the first chapter, where I present evidence for and arguments against its existence. I then counter these objections with further studies and discuss my findings at length. Inconsistencies also exist within di↵erent products of foreground maps as shown in the second chapter. Both chapters hint towards problems with the inherent assumptions about foreground spectra in component separation techniques. In the third chapter I explore statistics of foregrounds both in temperature and polarization. The conclusions, that foreground emission can be treated as a Gaussian process on certain scales, have positive implications for foreground simulations. Lastly, I present a method for improved treatment of polarized data on incomplete skies, which, when compared with state-of-the-art solutions performs better by orders of magnitude. In brief, in this thesis I highlight problems in our current treatment of Galactic foregrounds at low and high frequencies by concrete examples, I argue for studying foregrounds’ statistics and present such investigations, and further propose methods for the analysis of polarized data.