Does the motor run OK when connected to one of the original drivers ?

without detailed information or boards to test
I can't be 100 % sure but one diagram for the TB6560 I have
shows the decay mode set to 100% not the 0% you posted

Another possible issue could be the acive high verses active low setting in mach3

without the circuit diagrams of the two boards I don't know if the step signal passes through to the TB6560 step pin unchanged or if it is being inverted

if the step pulse has been inverted the IC will see a very wide step pulse with a very narrow gap between pulses and as a result miss pulses
try changing the step pulse to active low to see what effect it has

the TB6560 steps on the positive transition of the step pulse as it goes from 0V to +5V

John