What type of spindle/collet?

If it's ER, then it sounds like you're not tightening it enough. ER collets have to be tight or bits will pull out. There are torque settings for standard ER collet nuts, but typically it's as tight as you can get it with your hands and standard sized spanners, however that's assuming things are well anchored. On a freely spinning spindle, gauging tightness can be a bit tricky.