Nithin Gopalakrishnan Nair

Email ngopala2 (at) jhu (dot) edu

I am a second year PhD student in the Department of Electrical and Computer Engineering at VIU Lab , Johns Hopkins University, where I am advised by Prof. Vishal M. Patel . Prior to joining JHU, I obtained a Bachelor's and Master's dual degree (B.Tech& M.Tech) with a major in Electrical Engineering from Indian Institute of Technology, Madras . I did my Master's thesis was under the supervision of Prof AN Rajagopalan at the IPCV Lab, IIT Madras.

I work on problems in computer vision, more specifically on Image restoration, My research primarily focuses on Image restoration and deep generative modelling. Specifically I use generative models for multiple downstream tasks including Image synthesis, Image editing, Image translation as well as Image restoration

CV | Google Scholar | Github | Twitter | LinkedIn

2022 Updates
  • One paper accepted at IEEE FG 2023.
  • One paper accepted at IEEE WACV 2023.
  • Two papers accepted at ICIP 2022.
  • Research intern at Mistubishi Electric Research Labs, Summer 2022.
Research

2022

sym Unite and Conquer: Cross Dataset Multimodal Synthesis using Diffusion Models
Nithin Goplakrishnan Nair Wele Gedara Chaminda Bandara and Vishal M Patel.
Arxiv, 2022
pdf | abstract | Project page

Generating photos satisfying multiple constraints finds broad utility in the content creation industry. A key hurdle to accomplishing this task is the need for paired data consisting of all modalities (i.e., constraints) and their corresponding output. Moreover, existing methods need retraining using paired data across all modalities to introduce a new condition. This paper proposes a solution to this problem based on denoising diffusion probabilistic models (DDPMs). Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Since each sampling step in the DDPM follows a Gaussian distribution, we show that there exists a closed-form solution for generating an image given various constraints. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task through our proposed sampling strategy. We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints. We perform experiments on various standard multimodal tasks to demonstrate the effectiveness of our approach.

          
sym Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors
Kangfu Mei*,Nithin Goaplakrishnan Nair*, Vishal M Patel.
Arxiv, 2022
pdf | abstract | Project page

Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models

          
sym Remote Sensing Change Detection using Denoising Diffusion Probabilistic Models
Wele Gedara Chaminda Bandara,Nithin Goplakrishnan Nair and Vishal M Patel.
Arxiv, 2022
pdf | abstract | Project page

Human civilization has an increasingly powerful influence on the earth system, and earth observations are an invaluable tool for assessing and mitigating the negative impacts. To this end, observing precisely defined changes on Earth’s surface is essential, and we propose an effective way to achieve this goal. Notably, our change detection (CD) method proposes a novel way to incorporate the millions of off-the-shelf, unlabeled, remote sensing images available through different earth observation programs into the training process through denoising diffusion probabilistic models. We first leverage the information from these off-the-shelf, uncurated, and unlabeled remote sensing images by using a pre-trained denoising diffusion probabilistic model and then employ the multi-scale feature representations from the diffusion model decoder to train a lightweight CD classifier to detect precise changes. The experiments performed on four publically available CD datasets show that the proposed approach achieves remarkably better results than the state-of-the-art methods in F1, IoU and overall accuracy..

          
sym T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models
Nithin Goplakrishnan Nair and Vishal M Patel.
accepted at IEEE FG, 2023
pdf | abstract

Modern-day surveillance systems perform person recognition using deep learning-based face verification networks. Most state-of-the-art facial verification systems are trained using visible spectrum images. But, acquiring images in the visible spectrum is impractical in scenarios of low-light and nighttime conditions, and often images are captured in an alternate domain such as the thermal infrared domain. Facial verification in thermal images is often performed after retrieving the corresponding visible domain images. This is a well-established problem often known as the Thermal-to-Visible (T2V) image translation. In this paper, we propose a Denoising Diffusion Probabilistic Model (DDPM) based solution for T2V translation specifically for facial images. During training, the model learns the conditional distribution of visible facial images given their corresponding thermal image through the diffusion process. During inference, the visible domain image is obtained by starting from Gaussian noise and performing denoising repeatedly. The existing inference process for DDPMs is stochastic and time-consuming. Hence, we propose a novel inference strategy for speeding up the inference time of DDPMs, specifically for the problem of T2V image translation. We achieve the state-of-the-art results on multiple datasets. The code and pretrained models are publically available at http://github.com/Nithin-GK/T2V-DDPM.

          
sym AT-DDPM: Restoring Faces degraded by Atmospheric Turbulence using Denoising Diffusion Probabilistic Models
Nithin Gopalakrishnan Nair, Kangfu Mei and Vishal M.Patel .
accepted at IEEE WACV, 2023
pdf | abstract

Although many long-range imaging systems are designed to support extended vision applications, a natural obstacle to their operation is degradation due to atmospheric turbulence. Atmospheric turbulence causes significant degradation to image quality by introducing blur and geometric distortion. In recent years, various deep learning-based single image atmospheric turbulence mitigation methods, including CNN-based and GAN inversion-based, have been proposed in the literature which attempt to remove the distortion in the image. However, some of these methods are difficult to train and often fail to reconstruct facial features and produce unrealistic results especially in the case of high turbulence. Denoising Diffusion Probabilistic Models (DDPMs) have recently gained some traction because of their stable training process and their ability to generate high quality images. In this paper, we propose the first DDPM-based solution for the problem of atmospheric turbulence mitigation. We also propose a fast sampling technique for reducing the inference times for conditional DDPMs. Extensive experiments are conducted on synthetic and real-world data to show the significance of our model. To facilitate further research, all codes and pretrained models will be made public after the review process.

            @article{nair2022ddpm,
  title={AT-DDPM: Restoring Faces degraded by Atmospheric Turbulence using Denoising Diffusion Probabilistic Models},
  author={Nair, Nithin Gopalakrishnan and Mei, Kangfu and Patel, Vishal M},
  journal={arXiv preprint arXiv:2208.11284},
  year={2022}
}
          
sym NBD-GAP: Non-Blind Image Deblurring Without Clean Target Images
Nithin Gopalakrishnan Nair, Rajeev Yasarla, Vishal M. Patel.
accepted at ICIP, 2022
pdf | abstract

In recent years, deep neural network-based restoration methods have achieved state-of-the-art results in various image deblurring tasks. However, one major drawback of deep learning-based deblurring networks is that large amounts of blurry-clean image pairs are required for training to achieve good performance. Moreover, deep networks often fail to perform well when the blurry images and the blur kernels during testing are very different from the ones used during training. This happens mainly because of the overfitting of the network parameters on the training data. In this work, we present a method that addresses these issues. We view the non-blind image deblurring problem as a denoising problem. To do so, we perform Wiener filtering on a pair of blurry images with the corresponding blur kernels. This results in a pair of images with colored noise. Hence, the deblurring problem is translated into a denoising problem. We then solve the denoising problem without using explicit clean target images. Extensive experiments are conducted to show that our method achieves results that are on par to the state-of-the-art non-blind deblurring works.

          
          
sym A comparison of different atmospheric turbulence simulation methods for image restoration
Nithin Gopalakrishnan Nair, Kangfu Mei,and Vishal M. Patel.
accepted at ICIP, 2022
pdf | abstract

Atmospheric turbulence deteriorates the quality of images captured by long-range imaging systems by introducing blur and geometric distortions to the captured scene. This leads to a drastic drop in performance when computer vision algorithms like object/face recognition and detection are performed on these images. In recent years, various deep learning-based atmospheric turbulence mitigation methods have been proposed in the literature. These methods are often trained using synthetically generated images and tested on real-world images. Hence, the performance of these restoration methods depends on the type of simulation used for training the network. In this paper, we systematically evaluate the effectiveness of various turbulence simulation methods on image restoration. In particular, we evaluate the performance of two state-or-the-art restoration networks using six simulations method on a real-world LRFID dataset consisting of face images degraded by turbulence. This paper will provide guidance to the researchers and practitioners working in this field to choose the suitable data generation models for training deep models for turbulence mitigation. The implementation codes for the simulation methods, source codes for the networks and the pre-trained models are available at https://github.com/Nithin-GK/Turbulence-Simulations

                @article{nair2022comparison,
                    title={A comparison of different atmospheric turbulence simulation methods for image restoration},
                    author={Nair, Nithin Gopalakrishnan and Mei, Kangfu and Patel, Vishal M},
                    journal={arXiv preprint arXiv:2204.08974},
                    year={2022}
                  }
          

2021

sym Confidence Guided Network For Atmospheric Turbulence Mitigation
Nithin Gopalakrishnan Nair, Vishal M.Patel .
ICIP, 2021
pdf | abstract

Atmospheric turbulence can adversely affect the quality of images or videos captured by long range imaging systems. Turbulence causes both geometric and blur distortions in images which in turn results in poor performance of the subsequent computer vision algorithms like recognition and detection. Existing methods for atmospheric turbulence mitigation use registration and deconvolution schemes to remove degradations. In this paper, we present a deep learning-based solution in which Effective Nearest Neighbors (ENN) based method is used for registration and an uncertainty-based network is used for restoration. We perform qualitative and quantitative comparisons using synthetic and real-world datasets to show the significance of our work.

                @inproceedings{nair2021confidence,
                    title={Confidence guided network for atmospheric turbulence mitigation},
                    author={Nair, Nithin Gopalakrishnan and Patel, Vishal M},
                    booktitle={2021 IEEE International Conference on Image Processing (ICIP)},
                    pages={1359--1363},
                    year={2021},
                    organization={IEEE}
                  }
          
sym Confidence Guided Network For Atmospheric Turbulence Mitigation
Mahesh Mohan MR Nithin Gopalakrishnan Nair, AN Rajagopalan .
TIP, 2021
pdf | abstract

Deep Dynamic Scene Deblurring for Unconstrained Dual-Lens Cameras

                @article{mohan2021deep,
                    title={Deep dynamic scene deblurring for unconstrained dual-lens cameras},
                    author={Mohan, MR Mahesh and Nithin, GK and Rajagopalan, AN},
                    journal={IEEE Transactions on Image Processing},
                    volume={30},
                    pages={4479--4491},
                    year={2021},
                    publisher={IEEE}
                  }
          
Internships
  • MERL (Summer 2022)
    Topic: Diffusion Models
Footer Website Template taken from Yogesh Balaji © 2022 GitHub, Inc. Footer navigation Terms Privacy Security