Order-by-disorder and spin-orbital liquids in a distorted Heisenberg-Kitaev model

The microscopic modeling of spin-orbit entangled j=1/2 Mott insulators such as the layered hexagonal iridates Na2IrO3 and Li2IrO3 has spurred an interest in the physics of Heisenberg-Kitaev models. Here we explore the effect of lattice distortions on the formation of the collective spin-orbital states that include not only conventionally ordered phases but also gapped and gapless spin-orbital liquids. In particular, we demonstrate that in the presence of distortions, i.e., spatial anisotropies of the exchange couplings, conventionally ordered states are formed through an order-by-disorder selection, which is not only sensitive to the type of exchange anisotropy but also to the relative strength of the Heisenberg and Kitaev couplings. The spin-orbital liquid phases of the Kitaev limit - a gapless phase in the vicinity of spatially isotropic couplings and a gapped Z2 phase for a dominant spatial anisotropy of the exchange couplings - show vastly different sensitivities to the inclusion of a Heisenberg exchange. While the gapless phase is remarkably stable, the gapped Z2 phase quickly breaks down in what might be a rather unconventional phase transition driven by the simultaneous condensation of its elementary excitations.