Nithin Gopalakrishnan Nair

Selected Publications

1. 

   Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis

   Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal M. Patel, and Tim K. Marks

   In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2023

   Abstract Bib Paper Code

   Conditional generative models typically demand large annotated training sets to achieve high-quality synthesis. As a result, there has been significant interest in designing models that perform plug-and-play generation, i.e., to use a predefined or pretrained model, which is not explicitly trained on the generative task, to guide the generative process (e.g., using language). However, such guidance is typically useful only towards synthesizing high-level semantics rather than editing fine-grained details as in image-to-image translation tasks. To this end, and capitalizing on the powerful fine-grained generative control offered by the recent diffusion-based generative models, we introduce Steered Diffusion, a generalized framework for photorealistic zero-shot conditional image generation using a diffusion model trained for unconditional generation. The key idea is to steer the image generation of the diffusion model at inference time via designing a loss using a pre-trained inverse model that characterizes the conditional task. This loss modulates the sampling trajectory of the diffusion process. Our framework allows for easy incorporation of multiple conditions during inference. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution. Our results demonstrate clear qualitative and quantitative improvements over state-of-the-art diffusion-based plug-and-play models while adding negligible additional computational cost.

   @inproceedings{Nair_2023_ICCV,
     author = {Nair, Nithin Gopalakrishnan and Cherian, Anoop and Lohit, Suhas and Wang, Ye and Koike-Akino, Toshiaki and Patel, Vishal M. and Marks, Tim K.},
     title = {Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
     month = oct,
     year = {2023},
     pages = {20850-20860},
   }

2. 

   Unite and Conquer: Plug & Play Multi-Modal Synthesis Using Diffusion Models

   Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, and Vishal M Patel

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2023

   Abstract Bib Paper Code

   Generating photos satisfying multiple constraints finds broad utility in the content creation industry. A key hurdle to accomplishing this task is the need for paired data consisting of all modalities (i.e., constraints) and their corresponding output. Moreover, existing methods need retraining using paired data across all modalities to introduce a new condition. This paper proposes a solution to this problem based on denoising diffusion probabilistic models (DDPMs). Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Since each sampling step in the DDPM follows a Gaussian distribution, we show that there exists a closed-form solution for generating an image given various constraints. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task through our proposed sampling This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. 6070 strategy. We also introduce a novel reliability parameter that allows using different off-the-shelf diffusion models trained across various datasets during sampling time alone to guide it to the desired outcome satisfying multiple constraints. We perform experiments on various standard multimodal tasks to demonstrate the effectiveness of our approach. More details can be found at: https://nithingk.github.io/projectpages/Multidiff

   @inproceedings{nair2023unite,
     title = {Unite and Conquer: Plug \& Play Multi-Modal Synthesis Using Diffusion Models},
     author = {Nair, Nithin Gopalakrishnan and Bandara, Wele Gedara Chaminda and Patel, Vishal M},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     month = jun,
     pages = {6070--6079},
     year = {2023},
   }

3. 

   Diffuse-Denoise-Count: Accurate Crowd-Counting with Diffusion Models

   Yasiru Ranasinghe, Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, and Vishal M Patel

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Mar 2024

   Abstract Bib Paper Code

   Crowd counting is a key aspect of crowd analysis and has been typically accomplished by estimating a crowddensity map and summing over the density values. However, this approach suffers from background noise accumulation and loss of density due to the use of broad Gaussian kernels to create the ground truth density maps. This issue can be overcome by narrowing the Gaussian kernel. However, existing approaches perform poorly when trained with such ground truth density maps. To overcome this limitation, we propose using conditional diffusion models to predict density maps, as diffusion models are known to model complex distributions well and show high fidelity to training data during crowd-density map generation. Furthermore, as the intermediate time steps of the diffusion process are noisy, we incorporate a regression branch for direct crowd estimation only during training to improve the feature learning. In addition, owing to the stochastic nature of the diffusion model, we introduce producing multiple density maps to improve the counting performance contrary to the existing crowd counting pipelines. Further, we also differ from the density summation and introduce contour detection followed by summation as the counting operation, which is more immune to background noise. We conduct extensive experiments on public datasets to validate the effectiveness of our method. Specifically, our novel crowd-counting pipeline improves the error of crowd-counting by up to 6 percent on JHU-CROWD++ and up to 7 percent on UCF-QNRF. The project will be available at github.com/dylran/DiffuseDenoiseCount

   @inproceedings{ranasinghe2023diffuse,
     title = {Diffuse-Denoise-Count: Accurate Crowd-Counting with Diffusion Models},
     author = {Ranasinghe, Yasiru and Nair, Nithin Gopalakrishnan and Bandara, Wele Gedara Chaminda and Patel, Vishal M},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     month = mar,
     year = {2024},
   }

4. 

   DDPM-CD: Remote Sensing Change Detection using Denoising Diffusion Probabilistic Models

   Wele Gedara Chaminda Bandara, Nithin Gopalakrishnan Nair, and Vishal M Patel

   arXiv preprint arXiv:2206.11892, Mar 2022

   Abstract Bib Paper Code

   Human civilization has an increasingly powerful influence on the earth system, and earth observations are an invaluable tool for assessing and mitigating the negative impacts. To this end, observing precisely defined changes on Earth’s surface is essential, and we propose an effective way to achieve this goal. Notably, our change detection (CD) method proposes a novel way to incorporate the millions of off-the-shelf, unlabeled, remote sensing images available through different earth observation programs into the training process through denoising diffusion probabilistic models. We first leverage the information from these off-the-shelf, uncurated, and unlabeled remote sensing images by using a pre-trained denoising diffusion probabilistic model and then employ the multi-scale feature representations from the diffusion model decoder to train a lightweight CD classifier to detect precise changes. The experiments performed on four publically available CD datasets show that the proposed approach achieves remarkably better results than the state-of-the-art methods in F1, IoU and overall accuracy. Code and pre-trained models are available at: https : //github.com/wgcban/ddpm − cd

   @article{bandara2022ddpm,
     title = {DDPM-CD: Remote Sensing Change Detection using Denoising Diffusion Probabilistic Models},
     author = {Bandara, Wele Gedara Chaminda and Nair, Nithin Gopalakrishnan and Patel, Vishal M},
     journal = {arXiv preprint arXiv:2206.11892},
     year = {2022},
   }