The triangular drill for drilling square holes has been around for a long time. It relies on a floating chuck to allow the drill to wander around the hole while still being rotated.
Geometery is shown well on this page - http://www.maa.org/mathtourist/matht..._08_31_09.html
You can do more than square holes, but it's not really that commonly used, as rotary broaching is a more versatile technique.