Why do I need accuracy? I am afraid the best answer I can give is, in the words of Sir Edmund Hillary, "because it's there". In constructional hobbies, and this is my hobby, most people strive for some principle in what they make: elegance, simplicity, beauty etc.. For me, the principle is accuracy.
As far as expence goes, what is a reasonable price to pay for a CNC milling machine? If you regard Arc Euro Trade's KX1 at £2350 as reasonable, then I think that I can build something quite special for less than that - especially as I have controllers, stepper motors etc.
What will I make with it? Clocks, small but accurate tools, a co-axial brushless DC motor for a model aircraft, wax models for investment casting, you name it. How long will it take? A couple of months if I work on it all of my spare time, a year ot two if I only spend the time "She Who Must Be Obeyed" allows me, but probably somewhere in between. Will it be worth it? If you are in business the answer must be "Hell no!" but if you are a hobbiest then the answer is "Damn yes!"
Regarding the idea of using Peltier temperature control. After the normal suspects, backlash, vibration etc., thermal differences in the workpiece and machine contribute a lot to errors. Controlling the temperature of bits that get hot such as the spindle and table, and bits that are long such as the upright bits of the gantry with Peltier devices is cheap - about £5 for a 90W device and somewhat less than that for the bits to control it, a few pounds for heatsinks and tubing and perhaps £20 for one of the water cooled radiators sold to computer modders. Including fans and concertina hose to dump the stepper and spindle motor heat outside the case I think there should be some change from a couple of hundred pounds.