The symplectic vortex equations and Hamiltonian Gromov-Witten invariants (II)