Is there an easy way to fix this? as I can only think of shiming the X-Y table but that will be a nightmare.
You could try adjusting the tension of the base mounting bolts; on a small machine like yours it might flex the post enough to correct the tram. Are the semi circles left by the cutter the same side when machining left to right and right to left? If the side swaps (i.e. the cutter is only cutting on the leading edge, then it could be the whole machine is flexing with the cutting load) lighter finishing cuts should fix it.

BTW the correct way to test the tram is with a DTI mounted on arm a few inched away from the chuck, so it can be rotated aroung the table (use a parallel between the table and the dti to even out irregularities in the table surface)