Nothing wrong with M5's

Realistically you need 2x the nominal diameter in threaded length to ensure you have at lease 3 complete threads engaged. This is realistically the best you can get.

M5 coarse threads come with a 0.8mm pitch, M8s come with 1.25mm pitch. You are thus correct that in 3mm steel, even though you can thread it, you will have bugger all thread.

Now, I would suggest that you should space your screw holes out such that you can clamp the bits you want nice and square. One M8 will not do the same job as 2 M5's in that case.

Also think of how the loads will be transmitted. That is what should govern how many screws you need.