And since the spindle has no through hole i cant fit a collet chuck and draw bar,
Tool holding is going to be your biggest problem (apart from the spindle bearings failing because they won't take side load). You cant use a normal jacobs type chuck to hold an end mill; it simply will not grip it tight enough for safe use (been there, have the scars) . You'll need to find some way of fitting a collect chuck (i.e. a cheap chinese one) to the drill spindle. If your current spindle has a taper fitting you'll need to supplement it with a bolt up into the the spindle else it will fall off (once it has come off the first time it'll never fit as tight again).

Once you've sorted the above, find yourself a reasonable (4" or bigger) cross slide vice and you will be able to make fairly good light cuts in ally.