It's always best to eliminate play, but there are costs to that, as you're discovering.

Most accurate way to machine bearing bores so they're in perfect alignment, is ideally do them in one go, but for that design wouldn't be too easy.
However, as the double row bearing gets clamped in position, you could machine it's bore to allow the bearing to move around a bit, so when you finally assemble it, the bearings get lined up via the shaft, and it's just a case of clamping the doublerow in place.