Hazarapet Tunanyan

publications

1. 

   Preprint

   Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else

   Hazarapet Tunanyan, Dejia Xu, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi

   In Arxiv preprint 2023

   Abs arXiv Website

   Recent advances in text-to-image diffusion models have enabled the photorealistic generation of images from text prompts. Despite the great progress, existing models still struggle to generate compositional multi-concept images naturally, limiting their ability to visualize human imagination. While several recent works have attempted to address this issue, they either introduce additional training or adopt guidance at inference time. In this work, we consider a more ambitious goal: natural multi-concept generation using a pre-trained diffusion model, and with almost no extra cost. To achieve this goal, we identify the limitations in the text embeddings used for the pre-trained text-to-image diffusion models. Specifically, we observe concept dominance and non-localized contribution that severely degrade multi-concept generation performance. We further design a minimal low-cost solution that overcomes the above issues by tweaking (not re-training) the text embeddings for more realistic multi-concept text-to-image generation. Our Correction by Similarities method tweaks the embedding of concepts by collecting semantic features from most similar tokens to localize the contribution. To avoid mixing features of concepts, we also apply Cross-Token Non-Maximum Suppression, which excludes the overlap of contributions from different concepts. Experiments show that our approach outperforms previous methods in text-to-image, image manipulation, and personalization tasks, despite not introducing additional training or inference costs to the diffusion steps.

2. 

   CVPR

   Specialist Diffusion: Plug-and-Play Sample-Efficient Fine-Tuning of Text-to-Image Diffusion Models to Learn Any Unseen Style

   Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi

   In IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR) 2023

   Abs Paper Code Poster Website

   Diffusion models have demonstrated impressive capability of text-conditioned image synthesis, and broader application horizons are emerging by personalizing those pretrained diffusion models toward generating some specialized target object or style. In this paper, we aim to learn an unseen style by simply fine-tuning a pre-trained diffusion model with a handful of images (e.g., less than 10), so that the fine-tuned model can generate high-quality images of arbitrary objects in this style. Such extremely lowshot fine-tuning is accomplished by a novel toolkit of finetuning techniques, including text-to-image customized data augmentations, a content loss to facilitate content-style disentanglement, and sparse updating that focuses on only a few time steps. Our framework, dubbed Specialist Diffusion, is plug-and-play to existing diffusion model backbones and other personalization techniques. We demonstrate it to outperform the latest few-shot personalization alternatives of diffusion models such as Textual Inversion and DreamBooth, in terms of learning highly sophisticated styles with ultra-sample-efficient tuning. We further show that Specialist Diffusion can be integrated on top of Textual Inversion to boost performance further, even on highly unusual styles.

3. 

   BMVC

   C4Net: Contextual Compression and Complementary Combination Network for Salient Object Detection

   Hazarapet Tunanyan

   In British Machine Vision Conference (BMVC) 2021

   Abs arXiv Poster

   Deep learning solutions of the salient object detection problem have achieved great results in recent years. The majority of these models are based on encoders and decoders, with a different multi-feature combination. In this paper, we show that feature concatenation works better than other combination methods like multiplication or addition. Also, joint feature learning gives better results, because of the information sharing during their processing. We designed a Complementary Extraction Module (CEM) to extract necessary features with edge preservation. Our proposed Excessiveness Loss (EL) function helps to reduce false-positive predictions and purifies the edges with other weighted loss functions. Our designed Pyramid-Semantic Module (PSM) with Global guiding flow (G) makes the prediction more accurate by providing high-level complementary information to shallower layers. Experimental results show that the proposed model outperforms the state-of-theart methods on all benchmark datasets under three evaluation metrics