Quick question - is that actually an end-mill/slotting cutter?, or a (very common) 0.8mm drill bit?

Next, what spindle speed are you operating at?, and how high can you go? Coupled with that, what's your depth-of-cut, and what are your X/Y feed rates.

Looking at the image, I'd be concerned from the pattern traced by the cutter that you're getting a lot of tool deflection, which could be symptomatic of too slow a spindle speed and too high a feed rate. But there's people with more experience than me on here that can offer more substantial advice.