Unless the case is grounded, you will nearly always get some kind of voltage reading between the case and ground. If you put any bit of metal near something that is either switching power or containing an active inductor, and you will get power generated in that bit of metal.

The case has to be properly grounded to ensure that power is dealt with safely, and to minimise noise transmission.
All grounds should come back to a central star point, and only be connected at one end to prevent ground loops.