Hi John welcome to the forum!

As far as I know there isn't a setting for back-off distance, it should work that the axis will "back off" until the switch changes state. You would normally set the axis position of your choice at this point in your button VB homing code etc.

For instance, when I push the Ref-All-Home button on my setup, all axis go to the switches, they back off the switch, then they go to a specific place on the machine bed that I then use as my main home/zero point....I then program everything in CAD from that zero or WC etc. Works great.

What code do you have in your "Ref-All-Home" button?

You could have a software problem with the Mach3 install, so either a software issue in that case re-install Mach then the button code, make sure you save and exit when doing button code edits.

If its a hardware issue, is it possible you have an electrical noise problem (false signal etc), or could it be a bad/faulty switch?