I would advise against getting a small pulley as I heard the belt wear is greater. Also you may struggle to get enough teeth meshed on a very small pulley.

Only disadvantage I can think of with a bigger pulley is the moment of inertia is greater, but that's probably negligible on a mill.