According to the Sherline website:
http://www.sherline.com/setupins.htm
The manual imperial mill gives 0.050" as the distance of one turn. So this makes it 20 tpi, ie a pitch of 1.27mm. I also say sthat the Sherline steppers are 0.9 degrees step
http://www.sherline.com/CNCfaq.htm
ie, 400 steps per revolution = 1.27 mm which means for you, 314.9606 steps per mm.
Yours are set to 320 per mm which will give 1.2903 mm per revolution.
So if you programmed a 5 mm move (Mach 3 will think this is 1600 steps), and it moved 1600 steps, you would actually move 1600/314.9606 = 5.08 mm which is what I origionally suggested.
I am not famiiar with Mach 3. Is there a way of entering steps per mm fraction? Waht you really want is 320 per 1.27mm. If you used 315 steps per mm you would be 0.125mm out over a meter. According to the specs, the longest travel is the X axis with is 9 inches = 228mm, 315 steps per mm would make full travel 0.029 mm too long. Not very much! The answer for exact acuracy if there is such a thing is to routinely scale all cad files by 0.9987 and use 315 steps per mm or order and fit metric screws.
Does that help? Anyway, I'd love one. I have an old (but very good nick) Isel mill, extruded, about A4 cutting area, preloaded ball screws, recirculating bearings and servo driven, home brew software. But I've lost some garage space and something may have to give at some point, all very sad.

Tom