Can't help on the VFD front other than to say try find a proper manual or swap it out for decent one. Don't see any reason or advantage for using Modbus in DIY enviroment so I'd avoid this if possbile.

On the Rails and bed/steps then this sounds very much like the frame or gantry is twisted. It's affectively rotating the tool on it's axis as it moves along the twisted surface rails are mounted on.