As far as smoothness goes then the better you can get it the better the finished result especially if you are using a spray where the coat thickness will be a lot less than with a brush or roller.
320 may feel flat to your touch but will be more likely to show up with a final coat. If you have preped it properly to start with then any rub down between coats should be minimal.
Those rollers will do an ok job, used rollers like that myself quite a few rimes when customers wanted wooden boards which needed to be painted. Before I got the spray gun that was lol. Just remember not to put the paint on to thick & finish all your strokes in the same direction.