Is the feedrate within spec for the particular drill you're using?

So far you've tried-
0.06mm/rev
0.04mm/rev
0.05mm/rev

Looking at a basic polished cobalt drill (you are using polished drills?), the recommended feedrate should be about 0.1mm/rev, while a solid carbide is slightly higher depending on the alloy, and high-helix drills being higher yet at around 0.15mm/rev.