Like stepper drivers, the VFD can suffer terminal damage if you connect or disconnect the load when powered. Putting a kill switch in the spindle wiring is only appropriate if you believe there is a real risk of someone getting an electric shock from the spindle itself and don't care if the VFD gets wrecked to save them. If you believe such a risk is real then you shouldn't switch it on in the first place! Safety in this respect has more to do with adequate earthing and proper installation of the wiring. If you're still worried then never touch the powered spindle.
How long the spindle takes to run down to a stop (which is what I think you mean by the spindle 'down time') is not relevant in regard to a personal injury from the spinning tool since by the time you or someone else gets to hit the kill switch it will be too late.