Dynomotion KFlop could handle what you're trying to do, provided you understand the kinematics involved, and aren't averse to a bit C programming.

Mach could be used, but I think that would involve CAM being responsible for handling tool offsets/angles, as I don't think it can handle the tool offsests when you get into angling tools around.

I'm sure LinuxCNC might be able to handle it, but I've got no idea how much it can do without getting deep into custom programming.