Inelastic neutron scattering has been used to study the magnetoelastic excitations in the multiferroic manganite hexagonal YMnO3. An avoided crossing is found between magnon and phonon modes close to the Brillouin zone boundary in the (a,b) plane. Neutron polarization analysis reveals that this mode has mixed magnon-phonon character. An external magnetic field along the c axis is observed to cause a linear field-induced splitting of one of the spin-wave branches. A theoretical description is performed, using a Heisenberg model of localized spins, acoustic phonon modes, and a magnetoelastic coupling via the single-ion magnetostriction. The model quantitatively reproduces the dispersion and intensities of all modes in the full Brillouin zone, describes the observed magnon-phonon hybridized modes, and quantifies the magnetoelastic coupling. The combined information, including the field-induced magnon splitting, allows us to exclude several of the earlier proposed models and point to the correct magnetic ground state symmetry, and provides an effective dynamic model relevant for the multiferroic hexagonal manganites.