I think you're describing the machine cutting curves with straight line approximations, so for instance if you cut a circle it will come out as an n-sided polygon? If you could post the g-code we can immediately see if this is the problem.
I had that happen once with V-carve when I accidently changed a setting it decided to export the toolpath at a really poor resolution, so used several straight lines instead of arcs. I think it was the toolpath tolerance setting, so in V-carve try going to 'Edit' then 'Options' and change the toolpath tolerances to a very small number, say 0.001, then try again. Clicking 'Restore ALL Original Settings' may also work.