For deep hole drilling, you really need good lubrication, and good chip clearance.

I've got an aluminium part that I drill on the lathe using a 6.8mm drill and 30mm deep (about 4.5 dia x depth). If the flood coolant fails (even just getting knocked away from hitting the drill), the drill will be welded to the part and snapped within a couple parts.
Polished drills greatly reduce chip welding (I use Phantom 11.160's, which are a polished cobalt stub drill).

For good chip clearance, you really need to ensure you're getting a good chip by using a reasonable DOC, which you may struggle to do with your spindle.

Having just ran the figures based on the drills I use, the optimum speed in soft aluminium is around 4400RPM, with a DOC of 0.135 per revolution, which should result in a required spindle torque of 0.7NM for a sharp drill.
Even with a solid carbide drill, the max speed is 12500rpm, but with a slightly higher DOC (and subsequently higher torque requirement - I've only got my spreadsheet setup for 11.160 drills)

The key thing is DOC and torque. Your spindle should be pretty much constant torque regardless of speed, so if it can't manage the require torque at a few thousand RPM, then it's not likely to manage at a higher speed. You could set the VFD to give a torque boost at lower RPM (increase the max current), but you do risk overheating the windings, however for peck drilling the load will be intermittent.