my first instinct tells me that adding mass to the screw would lower the frequncy of the screws sympathetic resonance or just devide the screw into a higher frequency harmonic... it may be possible to tune the resonace out of the reach of the vibration causing it?
im guessing it would take a lot of work in either trail and error or some funky maths...you would lose some of your axis travel and getting a good solid contact between your mass and the screw would take a bit of doing

are you talking about resonance or whip?
the way i see it, resonance is a flex mass vibration type of thing (guitar string style)
whipping is the screw acting like a skipping rope

if your talking about whipping i would take a look at jonathan's revolving ball nut design