Contracting Transport Maps on Curved Spaces