Are you using the cncdrive Servoconfigurator3 software to test and debug the servos? https://www.cncdrive.com/downloads.html It sounds like ti from what you say.

The behaviour sounds like classic servo runaway which happens when the encoder signal is the wrong polarity relative to how the motor wants to spin. So in this case the drive tries to reduce the position error by going even further in the wrong direction. It doesn't know any better.

You should be able to talk to the drive without the motor power enabled ie provide the 5V but not the 130V. Then the s/w should see the drive and the encoder without it spinning off into the bushes. You should be able to turn the motor and check the operation and polarity of the encoder signal. It's a good idea to check the encoder signal makes some sort of sense before going any further.

With a DC brushed motor, you can't get the phases mixed up because there aren't any. But you can swap over the motor polarity by simply swapping the wires over. That's the 2 heavy wires from the drive to the motor. You'd think this should stop the runaway by changing the direction of the motor. Try that and see how you get on....

The motor wires don't need to be screened to work properly but it's good practice. Many of the problems people encounter seem to be caused by electrical noise, usually because of poor wiring practice. You don't need to be an expert but the methods are well understood and explained fairly well in places. I'm sure you will be able to find some good pointers if you look for them and we can always help you with suggestions.

You must be getting pretty close now!