I had large cast iron wheels like that on my Bridgeport. I couldn't get a decent response from the DMM servos until I removed them. Their moment of inertia was much bigger than the ballscrew itself and the reflected inertia of the table and vise, particularly on the X axis which had 2 of the buggers. As I have an MPG, turns out I never use the handwheels anyway but your solution would be a pragmatic middle way.