Let me just try and get my head around this...

If you set the spindle axis of rotation square to the bed you have an X, Y error as you move in the Z axis.

You estimate about 1mm in X, Y over the entire Z travel.

Also the Z axis dovetail slots are cut in the same metal part as the bearing surface that supports the sensitive drilling spindle.