This discussion is getting dangerously close to this design:
Rapid Changer
I made a prototype of that just to test the mechanism, but a bit bigger. It holds the holder reasonably well, but since it was only a test I used a random grade of steel which didn't harden much and I didn't grind it. I think it could work, but I doubt the holding force and torque transmission would be that great.
Attachment 6940Attachment 6941Attachment 6942Attachment 6943
I bought some EN19 steel to have a go at making a proper one...