Tend to agree with Eddy.

Part Fixturing needs to be perfect when flipping and the slightest error will be highlighted like your seeing. This is made worse in your case because your drilling which in it's self is not an easy thing to do with great accurecy without very carefull attention to detail. Really if you want accurate thru holes then you should be drilling under size from one side or both and then reaming thru from one side.

Personally I would make a Fixture Jig for the part when flipping and fasten the pre drilled side to the fixture jig with the drilled holes using short alignment pins. This way your only working from the fixture plate and because part is located by the holes and hole locations always known by referencing from the fixture plate it's much easier to align and get accurate. The pins in the Jig align the part perfectly and this will give you the best chance.

The Type of drill, length etc along with pre drilling etc and materail changes due to heat etc will all come into play so drilling accurate holes isn't an easything at best of times and with flipping then all new dimension comes into play.!!