Thank you, that's helpful. if the toolholder getting stuck is the biggest worry then I might try it once or twice to see how hard it is to remove them after they've been tightened up by some heavy cuts. I don't have to modify anything to test that. If you are right, as I suspect you will be, I'll look into boring out the spindle.
I wonder if that could be done in-situ with a series of drills the same diameter and different lengths? I'd prefer to not take the whole spindle assembly apart if I can avoid it.