We present measurements of the stellar mass functions (SMFs) of star-forming and quiescent galaxies to z = 4 using a sample of 95,675 Ks -selected galaxies in the COSMOS/UltraVISTA field. The SMFs of the combined population are in good agreement with previous measurements and show that the stellar mass density of the universe was only 50%, 10%, and 1% of its current value at z 0.75, 2.0, and 3.5, respectively. The quiescent population drives most of the overall growth, with the stellar mass density of these galaxies increasing as ρ(1 + z) since z = 3.5, whereas the mass density of star-forming galaxies increases as ρ(1 + z). At z > 2.5, star-forming galaxies dominate the total SMF at all stellar masses, although a non-zero population of quiescent galaxies persists to z = 4. Comparisons of the Ks -selected star-forming galaxy SMFs with UV-selected SMFs at 2.5 <z <4 show reasonable agreement and suggest that UV-selected samples are representative of the majority of the stellar mass density at z > 3.5. We estimate the average mass growth of individual galaxies by selecting galaxies at fixed cumulative number density. The average galaxy with log(M /M ) = 11.5 at z = 0.3 has grown in mass by only 0.2 dex (0.3 dex) since z = 2.0 (3.5), whereas those with log(M /M ) = 10.5 have grown by >1.0 dex since z = 2. At z <2, the time derivatives of the mass growth are always larger for lower-mass galaxies, which demonstrates that the mass growth in galaxies since that redshift is mass-dependent and primarily bottom-up. Lastly, we examine potential sources of systematic uncertainties in the SMFs and find that those from photo-z templates, stellar population synthesis modeling, and the definition of quiescent galaxies dominate the total error budget in the SMFs.