One problem is that even when they are not powered, stepper motors are quite "notchy" to turn which makes steady movement of a handwheel more difficult. It's possible but not quite as good - you need to get hold of a stepper motor and just turn the spindle to see what it's like. One possibility is to arrange some kind of clutch mechanism. Another alternative is to accept that you have to power up the control box, etc, and use an MPG - a box with a knob on, crudely - to control movement. Typically the MPG will let you select the axis you want to move and the step size you want and you then twiddle away to move an axis. Some people go as far as to fit "electronic handwheels" to replace the manual ones.

One other problem is that depending on how you do the conversion, you might lose the handwheel dials so you have no manual position readout. If you want to use the CNC system in purely manual mode, you won't be able to use the electronic position display unless you use something more sophisticated than simple stepper motors - unless you use the MPG approach.

It's also possible that what I am saying is more applicable to larger, heavier, machines and the problem might not be so bad on a mini-mill size machine - it would be good to hear from users of that style of machine for real experience.