GitLab

This is the project page for the paper The Missing Data Encoder: Cross-Channel Image Completion with Hide-And-Seek Adversarial Network by A.Dapogny, M.Cord and P.Perez, AAAI 2020

This work was done in collaboration between Datakalab (Paris, France), the LIP6 lab (Sorbonne Université, Paris, France) as well as Valeo.ai (Paris, France). If you use the code or results please use the following reference:

@article{
dapogny2019missing,
title={The Missing Data Encoder: Cross-Channel Image Completion with Hide-And-Seek Adversarial Network},
author={Dapogny, Arnaud and Cord, Matthieu and Perez, Patrick},
journal={AAAI},
year={2020}
}

This page provides code samples to ensure reproducibility, as well as some detailled results and side applications of MDE.

Introduction

This paper addresses the problem of image completion with deep neural networks, i.e. completing an image from fragments of it only. We introduces a general framework that we call the missing data encoder (MDE), which addresses many corner cases of image completion, namely colorization, inpainting, reverse inpainting (RI, also called outpainting), as well as more general subcases that we call random extrapolation (RE), and random extrapolation and colorization (REC), as illustrated below.

We propose an original architecture for MDE (and particularly RE and REC tasks) which employs adversarial training, perceptual losses, as well as a novel hide-and-seek (HnS) loss, an additional adversarial objective that consists in designing a separate network that tries to "seek" the original (non-masked) regions while the generator shall, in turn, "hide" them (see below).

See the paper for further explanation.

• Arxiv version (May 2019): https://arxiv.org/pdf/1905.01861.pdf
• AAAI 2020 version: coming soon!

Models trained with HnS loss have higher inception score, resulting in qualitatively better completed images, as shown below.

Experiments

This section introduces complementary experiments, either requested by the AAAI reviewers or that could not be included in the camera ready version due to lack of space.

Experiments with higher resolution images

The following figure presents application of MDE-REC (with S=0.2) on CelebA with larger images (2 times the resolution presented in the paper, so 192 x 192). Can you retrieve the non-masked regions on these? (Okay, the one on the right is quite easy!).

The image completion is indeed quite good in this case, though we did not modify the provided code beyond adding two layers in the generator (for the encoder/downsample part as well as the decoder/upsample part) with 32 channels, and the same for the discriminator/mask reconstruction network. We suspect that, to go higher resolution (which is not feasible on CelebA as most images does not have very high resolution) one will have to use several discriminator (and mask reconstruction networks) at multiple scales to facilitate training.

MDE with non-rectangular masks

The MDE-REC variant introduced in the paper can be further extended to irregular masks by introducing holes in the masks, either regular-which is analog to downsampling the non-masked regions, therefore which will be refered to as super resolution in what follows- or irregular, which amounts to considering noisy binary masks and will be refered to as denoising. In such a case, we introduce an additional hyperparameter p, which corresponds to the drop probability of each pixel of a non-masked region. Hyperparameters S (surface of non-masked regions) and p shall be carefully chosen adequately to the difficulty of the completion task. For instance, the following completed images and masks shows that images can be successfully completed on CelebA with S=0.2 and p=0.5 for a REC task:

Just as in the paper, we test the denoising/superres MDE-REC in the paired (same protocol for dropping pixels in train/test) and unpaired cases (different protocols, e.g. training in superres and testing for denoising). The results are encompassed in the following graphs:

Likewise, the denoising variant is also relevant for superres (high inception scores in the unpaired case) while the opposite is not true: henceforth the MDE-REC-denoising appears as a more general variant. This result is similar to this of Figure 7 in the paper, where a MDE-REC can be relevant for RE and colorization tasks whereas the opposite is not true.

Having fun with MDE

The possibilities of MDE are endless! First, we can apply MDE-RI (or outpainting) by successively cropping an image, applying MDE-RI to extrapolate from this crop, giving a larger image. Then, resize this crop and use MDE-RI again to generate an even larger image, and so on. Obviously, to do so, you have to either incorporate some sort of multi-scale scheme in your training, e.g. by augmenting the training data with zoom-in or training a separate MDE-RI model for each scale. As you can see in the following Figure, the original and (two-times) extrapolated images look quite different!

Another example is to first generate a random mask on a given image, then using either MDE-RE or MDE-REC (in the following Figure, we used MDE-REC with S=0.1) to complete the image. We then generate a new mask, mask out the newly completed image with this mask and apply MDE-REC again, and so on. The following figure shows 10 successive MDE-REC applications on several images. Here again, the completed faces quickly drifts away from the original ones!

Some insights and future work

Coming soon!