Aluminum is a bitch for such small holes. Of course you need a collet, not chuck. Even cheap chinese collet holds 0.01mm or better. And to tell you the truth when speaking about precision, you are most likely to loose precision from not cleaning properly your spindle or collet, than from the collet or spindle itself.
You definitely have to peck drill and make sure you blow away the chips. Chips no dust or rubbing.
Drills must be carbide and brand ones. Look at Kyocera on ebay, drillman1