Have you tried pushing the drill into the 4mm collet with it off the machine?? You can sometimes 'stretch' collets for odd sizes, I found that if you take the collet out of the spindle and pushed the cutter through it from the back it would give enough to get it in.

Did not work every time though...