Slowing the spindle down shouldn't be an issue if I keep the feedrate proportional, right? The chipload is the important thing I think. Still, maybe it will cut smoother if it is faster, I should try.

I might also simply need to try another cutting bit, perhaps this one is slightly damaged.