Most of my tooling is 1/8th, so I have a 1/8th collet - Simples!

Have you measured the tool stick-out before and after cutting? a 1/8th tool in a 3mm collet may just not be gripping properly, especially if it is not fully inserted in the first place..