Lucan, no offence, but your suggestion won't work.
Tapers rely on being firmly held together. With your suggestion, you would either have to ensure the tapers never fully tightened, meaning the toolholder will be loose and rattle in the spindle, or risk never being able to get the key in.