LinuxCNC will almost certainly handle the kinematics and you may want to use CNC-Toolkit, which is a free program for generating 5-axis toolpaths. You may have to do some programming to get linuxCNC to run the machine as it's not linear, but maybe not as there are a few existing things which may do what you need. Not linear and not full rotation implies you're making a hexapod (which linuxCNC supports directly), or just generally deriving the rotary motion from a linear actuator, i.e. ballscrew?