Deep Self-Dissimilarities as Powerful Visual Fingerprints

Idan Kligvasser; Tamar Rott Shaham; Yuval Bahat; Tomer Michaeli

Deep Self-Dissimilarities as Powerful Visual Fingerprints

Idan Kligvasser, Tamar Rott Shaham, Yuval Bahat, Tomer Michaeli

Technion - Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Features extracted from deep layers of classification networks are widely used as image descriptors. Here, we exploit an unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statistics across image scales, it turns out that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used as a powerful visual fingerprint. Particularly, we illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD as a loss function in training of image restoration networks, leads to results that are at least as photo-realistic as those obtained by GAN based methods, while not requiring adversarial training.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Pages	3939-3951
Number of pages	13
ISBN (Electronic)	9781713845393
State	Published - 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: 6 Dec 2021 → 14 Dec 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	5

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	6/12/21 → 14/12/21

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Access to Document

https://proceedings.neurips.cc/paper/2021/file/20479c788fb27378c2c99eadcf207e7f-Paper.pdf

Cite this

Kligvasser, I., Rott Shaham, T., Bahat, Y., & Michaeli, T. (2021). Deep Self-Dissimilarities as Powerful Visual Fingerprints. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 3939-3951). (Advances in Neural Information Processing Systems; Vol. 5). https://proceedings.neurips.cc/paper/2021/file/20479c788fb27378c2c99eadcf207e7f-Paper.pdf

Kligvasser, Idan ; Rott Shaham, Tamar ; Bahat, Yuval et al. / Deep Self-Dissimilarities as Powerful Visual Fingerprints. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. 2021. pp. 3939-3951 (Advances in Neural Information Processing Systems).

@inproceedings{3129175bbaf545288a65e00c1c643ea2,

title = "Deep Self-Dissimilarities as Powerful Visual Fingerprints",

abstract = "Features extracted from deep layers of classification networks are widely used as image descriptors. Here, we exploit an unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statistics across image scales, it turns out that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used as a powerful visual fingerprint. Particularly, we illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD as a loss function in training of image restoration networks, leads to results that are at least as photo-realistic as those obtained by GAN based methods, while not requiring adversarial training.",

author = "Idan Kligvasser and {Rott Shaham}, Tamar and Yuval Bahat and Tomer Michaeli",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

language = "الإنجليزيّة",

series = "Advances in Neural Information Processing Systems",

pages = "3939--3951",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, {Percy S.} and {Wortman Vaughan}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

Kligvasser, I, Rott Shaham, T, Bahat, Y & Michaeli, T 2021, Deep Self-Dissimilarities as Powerful Visual Fingerprints. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 5, pp. 3939-3951, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 6/12/21. <https://proceedings.neurips.cc/paper/2021/file/20479c788fb27378c2c99eadcf207e7f-Paper.pdf>

Deep Self-Dissimilarities as Powerful Visual Fingerprints. / Kligvasser, Idan; Rott Shaham, Tamar; Bahat, Yuval et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. 2021. p. 3939-3951 (Advances in Neural Information Processing Systems; Vol. 5).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Deep Self-Dissimilarities as Powerful Visual Fingerprints

AU - Kligvasser, Idan

AU - Rott Shaham, Tamar

AU - Bahat, Yuval

AU - Michaeli, Tomer

PY - 2021

Y1 - 2021

N2 - Features extracted from deep layers of classification networks are widely used as image descriptors. Here, we exploit an unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statistics across image scales, it turns out that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used as a powerful visual fingerprint. Particularly, we illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD as a loss function in training of image restoration networks, leads to results that are at least as photo-realistic as those obtained by GAN based methods, while not requiring adversarial training.

AB - Features extracted from deep layers of classification networks are widely used as image descriptors. Here, we exploit an unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statistics across image scales, it turns out that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used as a powerful visual fingerprint. Particularly, we illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD as a loss function in training of image restoration networks, leads to results that are at least as photo-realistic as those obtained by GAN based methods, while not requiring adversarial training.

UR - http://www.scopus.com/inward/record.url?scp=85131786990&partnerID=8YFLogxK

M3 - منشور من مؤتمر

T3 - Advances in Neural Information Processing Systems

SP - 3939

EP - 3951

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

Y2 - 6 December 2021 through 14 December 2021

ER -

Kligvasser I, Rott Shaham T, Bahat Y, Michaeli T. Deep Self-Dissimilarities as Powerful Visual Fingerprints. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. 2021. p. 3939-3951. (Advances in Neural Information Processing Systems).

Deep Self-Dissimilarities as Powerful Visual Fingerprints

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this