Kitaev anisotropy induces mesoscopic Z2 vortex crystals in frustrated hexagonal antiferromagnets

The triangular-lattice Heisenberg antiferromagnet (HAF) is known to carry topological Z2 vortex excitations which form a gas at finite temperatures. Here we show that the spin-orbit interaction, introduced via a Kitaev term in the exchange Hamiltonian, condenses these vortices into a triangular Z2 vortex crystal at zero temperature. The cores of the Z2 vortices show abrupt, soliton-like magnetization modulations and arise by a special intertwining of three honeycomb superstructures of ferromagnetic domains, one for each of the three sublattices of the 120 state of the pure HAF. This is an example of a nucleation transition, analogous to the spontaneous formation of magnetic domains, Abrikosov vortices in type-II superconductors, blue phases in cholesteric liquid crystals, and skyrmions in chiral helimagnets. As the mechanism relies on the interplay of geometric frustration and spin-orbital anisotropies, such vortex mesophases can materialize as a ground state property in spin-orbit coupled correlated systems with nearly hexagonal topology, as in triangular or strongly frustrated honeycomb iridates.