On Stability of Blow up Solutions to Critical Wave Maps Beyond the Equivariant Setting