We present a simulation-based inference framework using a convolutional neural network to infer dynamical masses of galaxy clusters from their observed 3D projected phase-space distribution, which consists of the projected galaxy positions in the sky and their line-of-sight velocities. By formulating the mass estimation problem within this simulation-based inference framework, we are able to quantify the uncertainties on the inferred masses in a straightforward and robust way. We generate a realistic mock catalogue emulating the Sloan Digital Sky Survey (SDSS) Legacy spectroscopic observations (the main galaxy sample) for redshifts z less than or similar to 0.09 and explicitly illustrate the challenges posed by interloper (non-member) galaxies for cluster mass estimation from actual observations. Our approach constitutes the first optimal machine learning-based exploitation of the information content of the full 3D projected phase-space distribution, including both the virialized and infall cluster regions, for the inference of dynamical cluster masses. We also present, for the first time, the application of a simulation-based inference machinery to obtain dynamical masses of around 800 galaxy clusters found in the SDSS Legacy Survey, and show that the resulting mass estimates are consistent with mass measurements from the literature.