Just measured my motors which are the 76 mm type and they measure 77.6mm so I still think you have the 57BYGH51-602A type.

Your waveforms look strange to me. The voltage across the current sense resistors should go between 0.5 V and - 0.5 V at 100% torque or 0.375 V at your 75% setting. Is the earth on your oscilloscope properly isolated from the board? If I remember rightly the isolators on these boards only isolate the signal lines not the earth. Are all three outputs the same?

Russell.