It's to do with the filter caps.

I'm guessing you're switching it with a double pole switch?
Try just switching the live wire, and see if things power up without any issues.