Do you have an oscilloscope?
That would be a sure fire way to find out if the encoder is working or not.

Failing that, you could test using a multimeter connected across the encoder pairs (A+/-, then B+/-, then make sure there is at least power on the Z+/- as stopping on the Z pulse will be near impossible) and randomly positioning the motor to see if the voltages invert. That at least tells you the encoder is doing something.
And if your multimeter has frequency measuring capability, you could spin the motor and see if if you get a plausible frequency reading.