StructureFlow_Image_Inpainting_via_Structure_Aware_Appearance_Flow.md

StructureFlow: Image Inpainting via Structure-aware Appearance Flow

Somodyuti Pal

With development of deep neural networks, reconstructing of hazy,distorted images or removing unwanted objects is a demanding application. Termed as Image Inpainting this was approached by diffusion based,patch based and deep-neural network framework based methods but failed to generate realistic structures of missing areas and gave disappointing results on non-repetitive patterns as it lacked in separating structure and texture of an image.

This Paper takes care of that by introducing two stages: 1.Structure reconstruction & 2. Texture generation. this two stages first recovers the missing structures and generate the reconstructed fine tuned data in last stage. Avoiding high frequency textures first edge images are used for structural guidance. This two stage network flow use Edge-preserved smooth methods to remove high-frequency textures while retaining sharp edges and low-frequency structures.

Structure Flow Architecture

The Proposed Method

• Edge preserved smooth images are used to reconstruct missing sections.the use of appearance flow ensures sample features from regions with similar structures. Then texture generator synthesizes high frequency details. And Gaussian sampling is used instead of bilinear sampling to expand the receptive field of sampling operation.The model consists of two parts: the structure reconstructor and the Texture generator

  • Structure Reconstructor - Let I_gt be the ground-truth image and S_gt be the edge-preserved smooth result of I_gt .Now processing of the structure reconstructor will be where M is the mask of the input image I_in. , a binarized matrix where 1 represents the missing region and 0 represents the background.

    The reconstruction loss of is defined as the distance between the predicted structures and the groundtruth structures .

    To achieve . GAN is applied and the Adversial loss is

    

    where is discriminator.Now the generator & discriminator is trained & optimized as

    

    where are regularized parameters.

  • Texture Generator - The texture generator takes and carrys on to generated vivid textures.this is

    .

    The adversial loss in texture generator will be now to establish long term dependency & a proper relation between different reegions the apearence Flow is introduced.Gaussian sampling is proposed to expand the receptive field.with a kernel size n this can be written as

  

  - **Sampling Correctness Loss** - This sampling correctnes loss playes a important key as it constraints the appearence flow .<img class="eqn-inline" src="https://latex.codecogs.com/svg.latex?\ V_{in} "> and  <img class="eqn-inline" src="https://latex.codecogs.com/svg.latex?\ V_{gt} "> be features generated by VGG19 layer.The sampling correctness loss calculate the relative
  cosine similarity between the ground truth features and the sampled features as:
  
    <img class="eqn-inline" src="https://latex.codecogs.com/svg.latex?\ L_c^t = \frac{1}{N} \sum_{x,y \in M } exp(-\frac{\mu (V_{x,y}^{gt},V_{x+\triangle x ,y+\triangle y}^{in})}{\mu_{x,y}^{max}})">
  
  
  
    <img class="eqn-inline" src="https://latex.codecogs.com/svg.latex?\ V_{x+\triangle x ,y+\triangle y}^{in} &space;">  is the sampled feature calculated byGaussian sampling and <img class="eqn-inline" src="https://latex.codecogs.com/svg.latex?\mu_{x,y}^{max}"> is a normalization term.
  

Experimental Results -

• Several resudial blocks with autoencoders were used to process the features.This model was tested Places2, CelebA,and Paris StreetView datasets. the structure reconstructor . and the texture generator . are trained separately using the edge-preserved image .

Qualitative analysis

• This model was compared with CA,Edge Connect and it outperformed both of them.the edge-preserved smooth images had the structures of images retrived well so,the model well balanced structure reconstruction and texture generation. Photo-realistic results are obtained hence. As these remove high frequency textures the proper tradeoff between structure reconstructor and the texture generator was acheived .

Relative performance

• Flow Ablation Study revealed Gaussian sampling & sampling correction loss effected real good as Bilinear sampling failed to capture long term correlations.

PSNR ,SSIM & SID compare

Conclusion -

Structure preservation and texture generator stages handled the inpaintaing challenge well.introducing appearance flow to to sample features from relative regions yield realistic image details,which improved the outcome in a geat manner. Experiments on multiple datasets shows the superior performance of the proposed network.

For code, visit this link.