Double check the wiring, but I can't think of a fault that would give you those kind of symptoms.

What does the X-axis DRO show on your controller?
Is it behaving how you'd expect (increasing when you jpg +, and decreasing when you jog -), or is it following what the motor is doing?

If the DRO is doing what you expect it should, then I'd try swapping drivers around, or at least swap step/dir wires with another axis, and see if the fault moves.