Very nice idea with the dust shoe. if you use some advanced CAM you could control all tool paths so no problem will arise. It seems even with simpler CAM it should work ok.

By the way i have a very similar dust shoe setup, only thing is that it does not slide like yours. Biggest problem i have encountered is when something chips and stucks in the section between the spindle and the vacuum hose. But my dust shoe seems not so deep like yours. So your should work better. i believe this dust shoe design will work better with short brush