Shouldn't bother the driver, as long as (as you say) you don't disconnect the motor when its actually powered up.

On my 4th axis, I isolate the mains feed to the servo drive by flicking a circuit breaker to save power etc when not connected. That's a simple and safe way to do it. No point having it powered up and doing nothing.

If your system requires all axes to be homed before you can use it, you might have an issue. On my Acorn system you can turn off the 4th axis and also disable homing for 4th axis anyway. Similarly, if you have an error / status signal back from the driver to your controller, you'd need to check it wouldn't cause an issue.