You may be missing steps. If you slow down the max feed rates in your CAM / g code, does the problem go away?

Also check that the maximum step rate from the controller to the stepper driver does not exceed what the stepper driver can handle. You may be able to reduce that somewhere in the settings - values such as 100kHz, 200kHz etc are common. You could try a reduced step rate if you are losing steps.

You should be able to calculate the number of pulses per mm rather than use trial and error. You know the ballscrew pitch and the pulley reduction ratio (if you have one), so the pulses per mm should be fairly simple to calculate directly.