Taking a decent cut of metal will of made a big difference.
If you don't take a deep enough cut, the heat builds up at the tip, leading to work hardening, and chip build up on the drill, which then reduces cutting effiency further.

It's best to look at the speeds/feeds for what you're using, and work out what the limiting factor of your machine is. Mostly in aluminium, spindle speed is the limiting factor.