Interpretable Neural Networks for Video Separation: Deep Unfolding RPCA With Foreground Masking

Boris Joukovsky, Yonina C. Eldar

Research output: Contribution to journalArticlepeer-review


We present two deep unfolding neural networks for the simultaneous tasks of background subtraction and foreground detection in video. Unlike conventional neural networks based on deep feature extraction, we incorporate domain-knowledge models by considering a masked variation of the robust principal component analysis problem (RPCA). With this approach, we separate video clips into low-rank and sparse components, respectively corresponding to the backgrounds and foreground masks indicating the presence of moving objects. Our models, coined ROMAN-S and ROMAN-R, map the iterations of two alternating direction of multipliers methods (ADMM) to trainable convolutional layers, and the proximal operators are mapped to non-linear activation functions with trainable thresholds. This approach leads to lightweight networks with enhanced interpretability that can be trained on limited data. In ROMAN-S, the correlation in time of successive binary masks is controlled with side-information based on $\ell _{1}$ - $\ell _{1}$ minimization. ROMAN-R enhances the foreground detection by learning a dictionary of atoms to represent the moving foreground in a high-dimensional feature space and by using reweighted- $\ell _{1}$ - $\ell _{1}$ minimization. Experiments are conducted on both synthetic and real video datasets, for which we also include an analysis of the generalization to unseen clips. Comparisons are made with existing deep unfolding RPCA neural networks, which do not use a mask formulation for the foreground, and with a 3D U-Net baseline. Results show that our proposed models outperform other deep unfolding networks, as well as the untrained optimization algorithms. ROMAN-R, in particular, is competitive with the U-Net baseline for foreground detection, with the additional advantage of providing video backgrounds and requiring substantially fewer training parameters and smaller training sets.

Original languageEnglish
Article number3336176
Pages (from-to)108-122
Number of pages15
JournalIEEE transactions on image processing : a publication of the IEEE Signal Processing Society
Early online date1 Dec 2023
StatePublished - 2024


  • Deep learning
  • deep unfolding
  • foreground detection
  • masked RPCA
  • video separation

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Interpretable Neural Networks for Video Separation: Deep Unfolding RPCA With Foreground Masking'. Together they form a unique fingerprint.

Cite this