It's called "chatter". A sort of sympathetic vibration and you've got it bad.

The easiest way to get chatter is big overhangs, reduce your overhangs.

Apart from that you can either increase the rigidity to damp it out or change the frequency either by adding flutes, as bogstandard suggests or by changing the RPM.

A radical spindle speed change before the finishing cut might help but chatter does tend to echo through on to the next cut.

With that much chatter, I agree, cutting uphill is probably a no-no.

Robin