Follow Rob's advice, all his observations are correct, he's being very gentle when he says your clamping looks a little flimsy, it ranks amongst the worst I've seen, don't muck about, get some metal clamps and use them properly.

You have a significant lack of rigidity for what you're trying to do, your roller ways combined with the low mass of your system are allowing the cutter to bounce around and move away from the work, because the deflection is greatest at the tip of the cutter the side of the cutter can engage the work avoided earlier, as your cut gets deeper the side of the cutter will grab the material avoided higher up the wall by earlier deflection.
Getting rid of the rollers running in channels and replacing them with good quality linear slides will give you some improvement but you may still come up short on mass and rigidity.