Bigger diameter cutters are *much* more rigid, so you want to use the biggest you can for a particular part. Within reason though as clearly a bigger diameter cutter will remove a greater volume of material, so the cutting force is higher for a given depth of cut. It's a balance ... I thought 6mm was good in general, but if you're only cutting say 5mm sheet then 4mm would probably be fine. I wouldn't advise anything bigger than 6mm for a router.

I've got some 0.5mm, 3mm and 4mm on the way, though they are two flute (oops).