Hank,
Cutting on the retract stroke will make the bar cut better as it is in tension. Your bar looks quite fat for the size of hole and could be rubbing near its bottom, it is quite important to get the clearances right but its ultimately a trade of against size and rigidity as the larger the hole the larger the cutter.
This is why I think most small holes are finished with a reamer.
Also somethings wrong if the drill is flexing all over the place, I've had it happen when the drill was sharpened off axis(ive a lot to learn about sharpening)
Peter