Get a 6-5mm collet.
Either the cutters you have are slightly less than 6mm, or the tolerance of the collet is off. Either way, I've found with ER collets, it's best to use the smallest one possible for whatever you're trying to hold.