Publications

2023

1. 

   Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis

   Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal M. Patel, and Tim K. Marks

   In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2023

   Abstract Bib Paper Code

   Conditional generative models typically demand large annotated training sets to achieve high-quality synthesis. As a result, there has been significant interest in designing models that perform plug-and-play generation, i.e., to use a predefined or pretrained model, which is not explicitly trained on the generative task, to guide the generative process (e.g., using language). However, such guidance is typically useful only towards synthesizing high-level semantics rather than editing fine-grained details as in image-to-image translation tasks. To this end, and capitalizing on the powerful fine-grained generative control offered by the recent diffusion-based generative models, we introduce Steered Diffusion, a generalized framework for photorealistic zero-shot conditional image generation using a diffusion model trained for unconditional generation. The key idea is to steer the image generation of the diffusion model at inference time via designing a loss using a pre-trained inverse model that characterizes the conditional task. This loss modulates the sampling trajectory of the diffusion process. Our framework allows for easy incorporation of multiple conditions during inference. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution. Our results demonstrate clear qualitative and quantitative improvements over state-of-the-art diffusion-based plug-and-play models while adding negligible additional computational cost.

   @inproceedings{Nair_2023_ICCV,
     author = {Nair, Nithin Gopalakrishnan and Cherian, Anoop and Lohit, Suhas and Wang, Ye and Koike-Akino, Toshiaki and Patel, Vishal M. and Marks, Tim K.},
     title = {Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
     month = oct,
     year = {2023},
     pages = {20850-20860},
   }

2. 

   Unite and Conquer: Plug & Play Multi-Modal Synthesis Using Diffusion Models

   Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, and Vishal M Patel

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2023

   Abstract Bib Paper Code

   Generating photos satisfying multiple constraints finds broad utility in the content creation industry. A key hurdle to accomplishing this task is the need for paired data consisting of all modalities (i.e., constraints) and their corresponding output. Moreover, existing methods need retraining using paired data across all modalities to introduce a new condition. This paper proposes a solution to this problem based on denoising diffusion probabilistic models (DDPMs). Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Since each sampling step in the DDPM follows a Gaussian distribution, we show that there exists a closed-form solution for generating an image given various constraints. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task through our proposed sampling This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. 6070 strategy. We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints. We perform experiments on various standard multimodal tasks to demonstrate the effectiveness of our approach. More details can be found at: https://nithingk.github.io/projectpages/Multidiff

   @inproceedings{nair2023unite,
     title = {Unite and Conquer: Plug \& Play Multi-Modal Synthesis Using Diffusion Models},
     author = {Nair, Nithin Gopalakrishnan and Bandara, Wele Gedara Chaminda and Patel, Vishal M},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     month = jun,
     pages = {6070--6079},
     year = {2023},
   }

3. 

   T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models

   Nithin Gopalakrishnan Nair, and Vishal M Patel

   In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), Jan 2023

   Abstract Bib Paper Code

   Modern-day surveillance systems perform person recognition using deep learning-based face verification networks. Most state-of-the-art facial verification systems are trained using visible spectrum images. But, acquiring images in the visible spectrum is impractical in scenarios of low-light and nighttime conditions, and often images are captured in an alternate domain such as the thermal infrared domain. Facial verification in thermal images is often performed after retrieving the corresponding visible domain images. This is a well-established problem often known as the Thermal-to-Visible (T2V) image translation. In this paper, we propose a Denoising Diffusion Probabilistic Model (DDPM) based solution for T2V translation specifically for facial images. During training, the model learns the conditional distribution of visible facial images given their corresponding thermal image through the diffusion process. During inference, the visible domain image is obtained by starting from Gaussian noise and performing denoising repeatedly. The existing inference process for DDPMs is stochastic and time-consuming. Hence, we propose a novel inference strategy for speeding up the inference time of DDPMs, specifically for the problem of T2V image translation. We achieve the state-of-the-art results on multiple datasets. The code and pretrained models are publically available at http://github.com/Nithin-GK/T2V-DDPM

   @inproceedings{nair2023t2v,
     title = {T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models},
     author = {Nair, Nithin Gopalakrishnan and Patel, Vishal M},
     booktitle = {2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)},
     pages = {1--7},
     month = jan,
     year = {2023},
     organization = {IEEE},
   }

4. 

   AT-DDPM: Restoring Faces Degraded by Atmospheric Turbulence Using Denoising Diffusion Probabilistic Models

   Nithin Gopalakrishnan Nair, Kangfu Mei, and Vishal M. Patel

   In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Jan 2023

   Abstract Bib Paper Code

   Although many long-range imaging systems are designed to support extended vision applications, a natural obstacle to their operation is degradation due to atmospheric turbulence. Atmospheric turbulence causes significant degradation to image quality by introducing blur and geometric distortion. In recent years, various deep learning-based single image atmospheric turbulence mitigation methods, including CNN-based and GAN inversionbased, have been proposed in the literature which attempt to remove the distortion in the image. However, some of these methods are difficult to train and often fail to reconstruct facial features and produce unrealistic results especially in the case of high turbulence. Denoising Diffusion Probabilistic Models (DDPMs) have recently gained some traction because of their stable training process and their ability to generate high quality images. In this paper, we propose the first DDPM-based solution for the problem of atmospheric turbulence mitigation. We also propose a fast sampling technique for reducing the inference times for conditional DDPMs. Extensive experiments are conducted on synthetic and real-world data to show the significance of our model. To facilitate further research, all codes and pretrained models are publically available at http://github.com/Nithin-GK/AT-DDPM

   @inproceedings{Nair_2023_WACV,
     author = {Nair, Nithin Gopalakrishnan and Mei, Kangfu and Patel, Vishal M.},
     title = {AT-DDPM: Restoring Faces Degraded by Atmospheric Turbulence Using Denoising Diffusion Probabilistic Models},
     booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
     month = jan,
     year = {2023},
     pages = {3434-3443},
   }

5. 

   Sar despeckling using a denoising diffusion probabilistic model

   Malsha V Perera, Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, and Vishal M Patel

   IEEE Geoscience and Remote Sensing Letters, Jan 2023

   Abstract Bib Paper Code

   Speckle is a type of multiplicative noise that affects all coherent imaging modalities including synthetic aperture radar (SAR) images. The presence of speckle degrades the image quality and can adversely affect the performance of SAR image applications such as automatic target recognition and change detection. Thus, SAR despeckling is an important problem in remote sensing. In this letter, we introduce SAR-DDPM, a denoising diffusion probabilistic model for SAR despeckling. The proposed method uses a Markov chain that transforms clean images into white Gaussian noise by successively adding random noise. The despeckled image is obtained through a reverse process that predicts the added noise iteratively, using a noise predictor conditioned on the speckled image. In addition, we propose a new inference strategy based on cycle spinning to improve the despeckling performance. Our experiments on both synthetic and real SAR images demonstrate that the proposed method leads to significant improvements in both quantitative and qualitative results over the state-of-the-art despeckling methods. The code is available at: https://github.com/malshaV/SAR_DDPM

   @article{perera2023sar,
     title = {Sar despeckling using a denoising diffusion probabilistic model},
     author = {Perera, Malsha V and Nair, Nithin Gopalakrishnan and Bandara, Wele Gedara Chaminda and Patel, Vishal M},
     journal = {IEEE Geoscience and Remote Sensing Letters},
     year = {2023},
     publisher = {IEEE},
   }

2022

1. 

   Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors

   Kangfu Mei, Nithin Gopalakrishnan Nair, and Vishal M Patel

   arXiv preprint arXiv:2212.07352, Jan 2022

   Abstract Bib Paper Code

   Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models. Our demo is https://kfmei.page/bi-noising/

   @article{mei2022bi,
     title = {Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors},
     author = {Mei, Kangfu and Nair, Nithin Gopalakrishnan and Patel, Vishal M},
     journal = {arXiv preprint arXiv:2212.07352},
     year = {2022},
   }

2. 

   DDPM-CD: Remote Sensing Change Detection using Denoising Diffusion Probabilistic Models

   Wele Gedara Chaminda Bandara, Nithin Gopalakrishnan Nair, and Vishal M Patel

   arXiv preprint arXiv:2206.11892, Jan 2022

   Abstract Bib Paper Code

   Human civilization has an increasingly powerful influence on the earth system, and earth observations are an invaluable tool for assessing and mitigating the negative impacts. To this end, observing precisely defined changes on Earth’s surface is essential, and we propose an effective way to achieve this goal. Notably, our change detection (CD) method proposes a novel way to incorporate the millions of off-the-shelf, unlabeled, remote sensing images available through different earth observation programs into the training process through denoising diffusion probabilistic models. We first leverage the information from these off-the-shelf, uncurated, and unlabeled remote sensing images by using a pre-trained denoising diffusion probabilistic model and then employ the multi-scale feature representations from the diffusion model decoder to train a lightweight CD classifier to detect precise changes. The experiments performed on four publically available CD datasets show that the proposed approach achieves remarkably better results than the state-of-the-art methods in F1, IoU and overall accuracy. Code and pre-trained models are available at: https : //github.com/wgcban/ddpm − cd

   @article{bandara2022ddpm,
     title = {DDPM-CD: Remote Sensing Change Detection using Denoising Diffusion Probabilistic Models},
     author = {Bandara, Wele Gedara Chaminda and Nair, Nithin Gopalakrishnan and Patel, Vishal M},
     journal = {arXiv preprint arXiv:2206.11892},
     year = {2022},
   }

3. 

   A Comparison of Different Atmospheric Turbulence Simulation Methods for Image Restoration

   Nithin Gopalakrishnan Nair, Kangfu Mei, and Vishal M. Patel

   In 2022 IEEE International Conference on Image Processing (ICIP), Jan 2022

   Abstract Bib Paper Code

   Atmospheric turbulence deteriorates the quality of images captured by long-range imaging systems by introducing blur and geometric distortions to the captured scene. This leads to a drastic drop in performance when computer vision algorithms like object/face recognition and detection are performed on these images. In recent years, various deep learning-based atmospheric turbulence mitigation methods have been proposed in the literature. These methods are often trained using synthetically generated images and tested on real-world images. Hence, the performance of these restoration methods depends on the type of simulation used for training the network. In this paper, we systematically evaluate the effectiveness of various turbulence simulation methods on image restoration. In particular, we evaluate the performance of two stateor-the-art restoration networks using six simulations method on a real-world LRFID dataset consisting of face images degraded by turbulence. This paper will provide guidance to the researchers and practitioners working in this field to choose the suitable data generation models for training deep models for turbulence mitigation. The implementation codes for the simulation methods, source codes for the networks and the pre-trained models are available at https://github.com/Nithin-GK/Turbulence-Simulations

   @inproceedings{9897969,
     author = {Gopalakrishnan Nair, Nithin and Mei, Kangfu and Patel, Vishal M.},
     booktitle = {2022 IEEE International Conference on Image Processing (ICIP)},
     title = {A Comparison of Different Atmospheric Turbulence Simulation Methods for Image Restoration},
     year = {2022},
     volume = {},
     number = {},
     pages = {3386-3390},
     doi = {10.1109/ICIP46576.2022.9897969},
   }

4. 

   NBD-GAP: Non-Blind Image Deblurring without Clean Target Images

   Nithin Gopalakrishnan Nair, Rajeev Yasarla, and Vishal M. Patel

   In 2022 IEEE International Conference on Image Processing (ICIP), Jan 2022

   Abstract Bib Paper

   In recent years, deep neural network-based restoration methods have achieved state-of-the-art results in various image deblurring tasks. However, one major drawback of deep learning-based deblurring networks is that large amounts of blurry-clean image pairs are required for training to achieve good performance. Moreover, deep networks often fail to perform well when the blurry images and the blur kernels during testing are very different from the ones used during training. This happens mainly because of the overfitting of the network parameters on the training data. In this work, we present a method that addresses these issues. We view the non-blind image deblurring problem as a denoising problem. To do so, we perform Wiener filtering on a pair of blurry images with the corresponding blur kernels. This results in a pair of images with colored noise. Hence, the deblurring problem is translated into a denoising problem. We then solve the denoising problem without using explicit clean target images. Extensive experiments are conducted to show that our method achieves results that are on par to the state-of-the-art non-blind deblurring works.

   @inproceedings{9897543,
     author = {Gopalakrishnan Nair, Nithin and Yasarla, Rajeev and Patel, Vishal M.},
     booktitle = {2022 IEEE International Conference on Image Processing (ICIP)},
     title = {NBD-GAP: Non-Blind Image Deblurring without Clean Target Images},
     year = {2022},
     volume = {},
     number = {},
     pages = {3431-3435},
     doi = {10.1109/ICIP46576.2022.9897543},
   }

2021

1. 

   Deep dynamic scene deblurring for unconstrained dual-lens cameras

   MR Mahesh Mohan, GK Nithin, and AN Rajagopalan

   IEEE Transactions on Image Processing, Jan 2021

   Abstract Bib Paper

   Dual-lens (DL) cameras capture depth information, and hence enable several important vision applications. Most present-day DL cameras employ unconstrained settings in the two views in order to support extended functionalities. But a natural hindrance to their working is the ubiquitous motion blur encountered due to camera motion, object motion, or both. However, there exists not a single work for the prospective unconstrained DL cameras that addresses this problem (so called dynamic scene deblurring). Due to the unconstrained settings, degradations in the two views need not be the same, and consequently, naive deblurring approaches produce inconsistent left-right views and disrupt scene-consistent disparities. In this paper, we address this problem using Deep Learning and make three important contributions. First, we address the root cause of view-inconsistency in standard deblurring architectures using a Coherent Fusion Module. Second, we address an inherent problem in unconstrained DL deblurring that disrupts scene-consistent disparities by introducing a memory-efficient Adaptive Scale-space Approach. This signal processing formulation allows accommodation of different image-scales in the same network without increasing the number of parameters. Finally, we propose a module to address the Space-variant and Image-dependent nature of dynamic scene blur. We experimentally show that our proposed techniques have substantial practical merit.

   @article{mohan2021deep,
     title = {Deep dynamic scene deblurring for unconstrained dual-lens cameras},
     author = {Mohan, MR Mahesh and Nithin, GK and Rajagopalan, AN},
     journal = {IEEE Transactions on Image Processing},
     volume = {30},
     pages = {4479--4491},
     year = {2021},
     publisher = {IEEE},
   }

2. 

   Confidence Guided Network For Atmospheric Turbulence Mitigation

   Nithin Gopalakrishnan Nair, and Vishal M. Patel

   In 2021 IEEE International Conference on Image Processing (ICIP), Jan 2021

   Abstract Bib Paper

   Atmospheric turbulence can adversely affect the quality of images or videos captured by long range imaging systems. Turbulence causes both geometric and blur distortions in images which in turn results in poor performance of the subsequent computer vision algorithms like recognition and detection. Existing methods for atmospheric turbulence mitigation use registration and deconvolution schemes to remove degradations. In this paper, we present a deep learning-based solution in which Effective Nearest Neighbors (ENN) based method is used for registration and an uncertainty-based network is used for restoration. We perform qualitative and quantitative comparisons using synthetic and real-world datasets to show the significance of our work.

   @inproceedings{9506125,
     author = {Nair, Nithin Gopalakrishnan and Patel, Vishal M.},
     booktitle = {2021 IEEE International Conference on Image Processing (ICIP)},
     title = {Confidence Guided Network For Atmospheric Turbulence Mitigation},
     year = {2021},
     volume = {},
     number = {},
     pages = {1359-1363},
     doi = {10.1109/ICIP42928.2021.9506125},
   }