What material is the fixture made from.?
What material is the part.?
What type of machine.? Mill or Router.
What type and material is the bed made from.?

Do you know if your part is flat or for that matter your fixture is truly flat or even the surface the fixture fastens down to is flat.? If any of them has any twist you will struggle to get a perfect fit.
Usually, you would surface the fixture so it's co-planer to the spindle, then you would machine the mating surface of the part if you need a close fit to the fixture.

If you have removed all burrs then I would suspect a twist in any of the areas mentioned above and most likely a little in each.