In that case drill a series of holes in piece of scrap then machine each one a slightly larger diameter (4.000, 4.001, 4.002) with a 4mm end mill. Then see which hole has the best fit with the pin and machine your plate that diamater.