Disentangling Structure and Appearance in ViT Feature Space

Narek Tumanyan, Omer Bar-Tal, Shir Amir, Shai Bagon, Tali Dekel

Research output: Contribution to journalArticlepeer-review


We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are "painted"with the visual appearance of their semantically related objects in a target appearance image. To integrate semantic information into our framework, our key idea is to leverage a pre-trained and fixed Vision Transformer (ViT) model. Specifically, we derive novel disentangled representations of structure and appearance extracted from deep ViT features. We then establish an objective function that splices the desired structure and appearance representations, interweaving them together in the space of ViT features. Based on our objective function, we propose two frameworks of semantic appearance transfer - "Splice", which works by training a generator on a single and arbitrary pair of structure-appearance images, and "SpliceNet", a feed-forward real-time appearance transfer model trained on a dataset of images from a specific domain. Our frameworks do not involve adversarial training, nor do they require any additional input information such as semantic segmentation or correspondences. We demonstrate high-resolution results on a variety of in-the-wild image pairs, under significant variations in the number of objects, pose, and appearance. Code and supplementary material are available in our project page: splice-vit.github.io.

Original languageEnglish
Article number11
Number of pages16
JournalACM Transactions on Graphics
Issue number1
Early online date30 Nov 2023
StatePublished - Feb 2024

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Disentangling Structure and Appearance in ViT Feature Space'. Together they form a unique fingerprint.

Cite this