Ahh, I just looked at the sheet I cut the part from and it seems to have a better finish on it. I used conventional milling as GWizzard suggested but it looks like climb is better?

I'd still like to know why it is okay on the other axis though and what causes it to look bad just on one of the passes.