Elliptic flow of $D^0$, $D^+$, $D^+_s$ and $\Lambda^+_c$ hadrons