Okay I have had another play.

I secured my Z axis properly to the X axis(!) and also made a kind of extender tube to get the vacuum suction closer to the work piece. I then tried a cut, but this time I reduced the spindle speed. I used GWizard to receommend a feed and speed and I noticed that actually it was giving me a smaller chipload than recomended. I suppose it was trying to keep tool deflection as low as possible but I found I could reduce the spindle RPM and still keep deflection below the problem area. GWizard sugested (for a 2flute 3.17mm bit) 1mm passes with 2000mm/min and 22,000RPM. I reduced this to 17,000RPM and kept the rest the same.

The cut is clean, but does have a slight texture, is this about as good as you guys are getting or should I still aim for higher?