Which mini lathe do you have? Anything obvious burnt on the driver board, picture? Might be able to help...

For HSS 800rpm sounds about right for that diameter. As you found the fly cutter being inherently imbalanced causes a lot of vibration which will very quickly wear out the spindle bearings at high speed. Even if you did balance the fly cutter it still needs to be operated at low speed as for the speed of the tip of the tool to be correct you have to lower the rpm as the diameter is increased.

What you're describing is the spindle being out of 'tram', so like you say the axis of rotation of the spindle is not orthogonal to the plane defined by the X and Y axis motion. Does it matter? If you're using a large diameter cutter and trying to get a very good finish then yes it does since the cut will be ever so slightly concave. Depends how far out it is...to measure it on a mill you would put a dial indicator in the spindle, facing down to touch the bed, mounted on the end of a reasonably long bar attached radially. Swing the indicator round and use it to measure the height of the bed at 4 points of the circle (12,3,6 and 9 o'clock). By comparing the readings you can find the gradient and thus how far out the spindle is and adjust accordingly (shims etc). That gets the spindle orthogonal to the machine bed. That's great on a mill but it's not a lot of use on the router unless the bed is already surfaced to get it parallel to X and Y. So you need to surface the bed first then indicate off it... but it will never be perfect due to the initial surfacing being slightly uneven due to the problem you're trying to correct! I'm sure it will be plenty near enough though as I bet most people don't bother with this at all. Unless it's out by a lot you only notice it on very large diameter cutters.

Edit: Also if you haven't already then grinding a reasonable radius on the fly cutter tip should improve the finish. Not too great though as it will increase the cutting force.