This discussion is getting dangerously close to this design:

Rapid Changer

I made a prototype of that just to test the mechanism, but a bit bigger. It holds the holder reasonably well, but since it was only a test I used a random grade of steel which didn't harden much and I didn't grind it. I think it could work, but I doubt the holding force and torque transmission would be that great.

Click image for larger version.  Name: DSC09318 (Large).JPG  Views: 2222  Size: 126.6 KB  ID: 6940Click image for larger version.  Name: DSC09319 (Large).JPG  Views: 2637  Size: 119.1 KB  ID: 6941Click image for larger version.  Name: DSC09320 (Large).JPG  Views: 4371  Size: 96.3 KB  ID: 6942Click image for larger version.  Name: DSC09321 (Large).JPG  Views: 2917  Size: 116.3 KB  ID: 6943

I bought some EN19 steel to have a go at making a proper one...