do you see the same behaviour (moving in & out from the workpiece) if you mount a DTI to your cross slide?
just trying to rule out spindle play as well