It sounds like the inverter isn't configured correctly for the motor.

Personally, given the space limitations, I'd consider fitting a servo motor and drive, as you get a lot more performance in a smaller size.