Using this info;

http://www.aliexpress.com/item/nema-...499066426.html
http://www.fozztexx.com/CNC/HY-TB3DV-M_3Axis_Driver.pdf (this is for 3 axis but I'm sure it still applies)

I think as a starting point you should set the DIP switches to this (in order 1 to 6, OFF OFF OFF ON OFF ON )
The motor wiring sounds okay, you have it in 'high torque' mode.
Follow the Mach3 settings in the document.