Colorful image reconstruction from neuromorphic event cameras with biologically inspired deep color fusion neural networks

Hadar Cohen-Duwek, Elishai Ezra Tsur

Research output: Contribution to journalArticlepeer-review

Abstract

Neuromorphic event-based cameras communicate transients in luminance instead of frames, providing visual information with a fine temporal resolution, high dynamic range and high signal-to-noise ratio. Enriching event data with color information allows for the reconstruction of colorful frame-like intensity maps, supporting improved performance and visually appealing results in various computer vision tasks. In this work, we simulated a biologically inspired color fusion system featuring a three-stage convolutional neural network for reconstructing color intensity maps from event data and sparse color cues. While current approaches for color fusion use full RGB frames in high resolution, our design uses event data and low-spatial and tonal-resolution quantized color cues, providing a high-performing small model for efficient colorful image reconstruction. The proposed model outperforms existing coloring schemes in terms of SSIM, LPIPS, PSNR, and CIEDE2000 metrics. We demonstrate that auxiliary limited color information can be used in conjunction with event data to successfully reconstruct both color and intensity frames, paving the way for more efficient hardware designs.

Original languageEnglish
Article number036001
JournalBioinspiration and Biomimetics
Volume19
Issue number3
DOIs
StatePublished - 1 May 2024

Keywords

  • Benchmarking
  • Equipment Design
  • Image Processing, Computer-Assisted
  • Neural Networks, Computer
  • Signal-To-Noise Ratio
  • event-based vision
  • image reconstruction
  • neuromorphic camera

All Science Journal Classification (ASJC) codes

  • Engineering (miscellaneous)
  • Molecular Medicine
  • Biophysics
  • Biochemistry
  • Biotechnology

Fingerprint

Dive into the research topics of 'Colorful image reconstruction from neuromorphic event cameras with biologically inspired deep color fusion neural networks'. Together they form a unique fingerprint.

Cite this