Clearly something in the system is non-linear. Have you measured that the pitch of your ballscrew is constant along it's length?

Have you measured that DRO error in both directions? What's the Backlash?

Otherwise I don't know how it could be anything but a broken encoder or some screwy software based correction.