I had one like that years ago and went for the light sanding option. You should be able to hold the pulley shaft end in a drill and wrap a strip of sandpaper around the journal. If you have a micrometer then aim for 11.97 - 12.00. If not then go by feel.
You will most likely need to take this apart a few times during building plus future maintenance so I wouldn't go the shrink fit route.

Given that bearings are machined very accurately you would think the shaft tolerance would be straightforward to hit.