Face0: Instantaneously Conditioning a Text-to-Image Model on a Face

Dani Valevski, Danny Lumen, Yossi Matias, Yaniv Leviathan

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

We present Face0, a novel way to instantaneously condition a text-to-image generation model on a face without any optimization procedures such as fine-tuning or inversions. We augment a dataset of annotated images with embeddings of the included faces and train an image generation model on the augmented dataset. Once trained, our system is practically identical at inference time to the underlying base model, and is therefore able to generate face-conditioned images in just a couple of seconds. Our method achieves pleasing results, is remarkably simple, extremely fast, and equips the underlying model with new capabilities, like controlling the generated images both via text or via direct manipulation of the input face embeddings. In addition, when using a fixed random vector instead of a face embedding from a user supplied image, our method essentially solves the problem of consistent character generation across images. Finally, our method decouples the model's textual biases from its biases on faces. While requiring further research, we hope that this may help reduce biases in future text-to-image models.

Original languageEnglish
Title of host publicationProceedings - SIGGRAPH Asia 2023 Conference Papers, SA 2023
EditorsStephen N. Spencer
ISBN (Electronic)9798400703157
DOIs
StatePublished - 10 Dec 2023
Externally publishedYes
Event2023 SIGGRAPH Asia 2023 Conference Papers, SA 2023 - Sydney, Australia
Duration: 12 Dec 202315 Dec 2023

Publication series

NameProceedings - SIGGRAPH Asia 2023 Conference Papers, SA 2023

Conference

Conference2023 SIGGRAPH Asia 2023 Conference Papers, SA 2023
Country/TerritoryAustralia
CitySydney
Period12/12/2315/12/23

Keywords

  • diffusion models
  • image editing

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design
  • Software
  • Computer Vision and Pattern Recognition

Fingerprint

Dive into the research topics of 'Face0: Instantaneously Conditioning a Text-to-Image Model on a Face'. Together they form a unique fingerprint.

Cite this