Competing lattice instability and magnetism on the surface of kagome metals

Only a few magnetic kagome materials exhibit lattice instabilities among the large kagome material array. In this work, we find that kagome magnets RMn6Sn6 (R = rare-earth elements) and their nonmagnetic counterparts RV6Sn6 exhibit an intriguing interplay between magnetism and lattice dynamics on their surfaces. Notably, RV6Sn6 surfaces terminated by kagome layers demonstrate pronounced lattice instabilities, manifesting as 3×3 and 2×2 surface charge orders (SCOs). These instabilities are absent on corresponding magnetic materials RMn6Sn6. Here, the SCO is suppressed by magnetism. Otherwise, surface distortions would significantly reduce the spin polarization, elevating the energy via Hund's rule. Thus, SCOs are energetically unfavorable on magnetic surfaces. The competition of magnetism and lattice instability is further substantiated by observing SCOs on V-substituted RMn6Sn6 kagome surfaces, contrasting with their absence on Mn-substituted RV6Sn6 surfaces. Our findings reveal unexpected surface instability and profound spin-lattice coupling in these kagome magnets, highlighting the complex dynamics hidden within magnetic materials.