I've just found this quote from John S. a long time ago:

One area where spinning ball nuts come into their own is on a CNC
lathe. ... a spin off is if you also fit a bevel gear to the nut block you can then drive this with another bevel mounted on a disengageable handwheel. This way you can
get to use a CNC lathe in manual mode without having to rely on
standing at one end turning the ball screw.
(Source: http://groups.yahoo.com/group/CAD_CA.../message/70038)

Backdriving could still be an issue, but you could add some friction to the handle and/or a lock easily enough to combat that. That's tempted me to convert my lathe ... I could probably still use my design for the router, perhaps with better (tapered roller maybe) bearings as the forces are likely to be greater. The tricky bit will be fitting the ballnut into the cross slide.