A few years back there were some articles about adding rigidity and mass to the small X1 mill by filling the hollow square column and also the base casting with epoxy concrete. Using an existing structure gives you a good starting point. You could also use standard aluminium or steel sections and fill those with epoxy-concrete, I believe this approach is used in industrial machines.

I think using lumps of granite will be fraught with problems - they are only finished flat on one face I guess so you couldn't rely on the edges being square or parallel. They are heavy to handle, hard to drill, and not strong in tension.