From the third photo it looks like you have issues with rigidity. Your machine has unsupported rails on all three axes, so this is shouldn't come as a surprise. That shouldn't mean you can't make this part though, just makes it more of a challenge.

Is the tool plunging too fast? Ideally it shouldn't much at all.

Have you checked the runout of the spindle? With such a small tool this may have an effect.