Whatever taper you go with you'll have the option to add collet chuck - ER Milling Collet Chucks - Arc Euro Trade
I haven't needed to remove the ER32 collet chuck on my manual mill for several weeks.
Or you could add a quick release system if you want it, my CNC has a Coventry Easy Change - Easy Change (Coventry Toolholders) System | Coventry Toolholders - Standard and Bespoke Tooling Systems
The adaptor and holders are expensive but it will save you time on jobs with several tool changes.
Other toolholding options are available ;-)
Regards,
Nick