Ignoring the probing code for a while, is there any reason that the probe switch distance should not be as short as possible, maybe 5mm?

I have mine built at 20mm but having sat and watched it today, it looks like 5mm would save a lot of Z axis travel time.

Is there any reason for a bigger gap or is smaller always better?