Your slot drill is bound to chew up the metal directly below it's centre, it is just something they do. A milling cutter has no cut towards the middle so it can leave a fine finish below. They all have their tricks.

However a milling cutter can only plunge vertically so far before it's non-cutting middle bottoms out, so you have to move sideways as you move downwards to remove that middle.

Well, either that or restrict your Z travel to what it can manage