On the Robustness of Normalizing Flows for Inverse Problems in Imaging

Seongmin Hong¹, Inbum Park¹ , Se Young Chun^1,2

¹Dept. of Electrical and Computer Engineering,
²INMC, Interdisciplinary Program in AI
Seoul National University, Republic of Korea

ICCV 2023

[Paper] [supp] [arXiv] [github] [bibTeX]

Unintended severe artifacts occur in conditional normalizing flows for inverse problems in imaging, which hinder their effectiveness.

Conditional normalizing flow: A generative model w/ fastness, diversity.

Inverse Problems in Imaging: Super-resolution, low-light enhancement, etc.

We found that these artifacts are caused by out-of-distribution (OOD) conditional inputs inducing "exploding inverses" in the conditional affine coupling layer (Aff).

OOD: Conditional input (i.e., y) whose embedding (i.e., g_θ (y)) has a large Mahalanobis distance [Lee et al., NeurIPS 2018] from the training set.

Exploding inverse: Originally reported in unconditional normalizing flows [Behrmann et al., AISTATS 2021], but the mechanism is different from the conditional normalizing flow.

Exploding inverse amplifies the magnitude of features, resulting in saturated colors (colorful artifacts) and NaN pixels (black region).

We revealed the origin of the errors: The slope (s) of the affine should have an upper bound s_u for training. s ≪ s_u makes the exploding inverse, and this happens when y is OOD.

We propose a remark to avoid the exploding inverse. We then propose a remedy to avoid the exploding inverse: use another one but affine, following Remark 2.

Another one: B-spline [Hong et al., AAAI 2023], or the modified RQ-spline (the figure).

To validate our findings, we conducted experiments on a toy dataset,

super-resolution space generation,

and low-light image enhancement.

Abstract

Conditional normalizing flows can generate diverse image samples for solving inverse problems. Most normalizing flows for inverse problems in imaging employ the conditional affine coupling layer that can generate diverse images quickly. However, unintended severe artifacts are occasionally observed in the output of them. In this work, we address this critical issue by investigating the origins of these artifacts and proposing the conditions to avoid them. First of all, we empirically and theoretically reveal that these problems are caused by “exploding inverse” in the conditional affine coupling layer for certain out-of-distribution (OOD) conditional inputs. Then, we further validated that the probability of causing erroneous artifacts in pixels is highly correlated with a Mahalanobis distance-based OOD score for inverse problems in imaging. Lastly, based on our investigations, we propose a remark to avoid exploding inverse and then based on it, we suggest a simple remedy that substitutes the affine coupling layers with the modified rational quadratic spline coupling layers in normalizing flows, to encourage the robustness of generated image samples. Our experimental results demonstrated that our suggested methods effectively suppressed critical artifacts occurring in normalizing flows for super-resolution space generation and low-light image enhancement.

BibTeX

 
@InProceedings{Hong_2023_ICCV,
    author    = {Hong, Seongmin and Park, Inbum and Chun, Se Young},
    title     = {On the Robustness of Normalizing Flows for Inverse Problems in Imaging},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {10745-10755}
}