Problem with using nylon wheels, is you risk losing rigidity due to flex in the wheels.

I wouldn't worry too much about running hardened bearings on an alloy extrusion. Provided you keep them clean and spinning, you shouldn't experience any major wear. You're only likely to experience major wear if things start sliding.