Lovish Chum

l......@columbia.edu

I am a Senior AI Engineer at Walgreens AI Labs where I am the part of Search and Recommendation team. Previously, I was an Machine Learning Engineer II at Amazon Web Services (AWS) - AI where I was the part of Rekognition team. I aim to create buisness impact through my scientific and engineering skills in computer vision and language models (especially LLMs).

Earlier I was a Research Asosicate at Dept. of Electrical Engineering, Columbia University with Prof. Shih-Fu Chang. I received MS(by research/thesis track) in Computer Science from Fu Foundation School of Engineering at Columbia University, where I worked on future scene graph prediction with Carl Vondrick.

From 2017 to 2019, I worked at CVIT where I worked with CV Jawahar and Vineeth N Balasubhramanian. Even earlier, I was an undergrad at IIT Kanpur.

I am always looking for opportunities. Welcome to contact via e-mail if you think it can be a good fit.

GitHub  /  LinkedIn

News

• Jan 2024: Paper on cross-modal event relations accepted at AAAI 2024
• Nov 2023: Joined Walgreens AI Lab. Excited to build the visual search engine for Walgreens website from scratch!
• Mar 2023: Paper on video representation learning accepted at ICASSP 2023
• Jun 2022: Paper on multi-modal event graphs available on arXiV
• Jan 2022: Joined Amazon Web Services (AWS) - AI as an ML Engineer in the Rekognition team
• Mar 2021: Paper accepted at EMNLP Findings 2021
• Mar 2021: Joined DVMM, Columbia University as Staff Associate I. Working with Prof. Shih-Fu Chang
• Oct 2020: Talk on Scene Graph Generation at CV Lab, Columbia
• Sep 2019: Joining Columbia University as a Masters of Computer Science student in Fall 2019
• Jul 2019: Talk on Linear Algebra for Machine Learning at IIIT-H ML Summer School
• May 2019: Survey paper accepted in Journal of Indian Institute of Science
• Sep 2018: Attending Machine Learning Summer School 2018 in Madrid

Recent Projects/Publications

Beyond Grounding: Extracting Fine-Grained Event Hierarchies across Modalities

Understanding how events described or shown in multimedia content relate to one another is a critical component to developing robust artificially intelligent systems which can reason about real-world media. While much research has been devoted to event understanding in the text, image, and video domains, none have explored the complex relations that events experience across domains. For example, a news article may describe a “protest” event while a video shows an “arrest” event. Recognizing that the visual “arrest” event is a subevent of the broader “protest” event is a challenging, yet important problem that prior work has not explored. In this paper, we propose the novel task of MultiModal Event Event Relations to recognize such cross-modal event relations. We contribute a large-scale dataset consisting of 100k video-news article pairs, as well as a benchmark of densely annotated data. We also propose a weakly supervised multimodal method which integrates commonsense knowledge from an external knowledge base (KB) to predict rich multimodal event hierarchies. Experiments show that our model outperforms a number of competitive baselines on our proposed benchmark. We also perform a detailed analysis of our model’s performance and suggest directions for future research.

Multimodal Event Graphs: Towards Event Centric Understanding of Multimodal World

Combining simple concepts to form structured thoughts and decomposing complex concepts into their constituents is one key characteristic of human cognition. In this work, we extract video representations by combining multi-scale processing with compositional constraints, i.e., we constrain the latent space created by the network so that coarse grained video features are composed from a set of fine-grained video features using simple functions. We integrate the proposed constraints in a state-of-the-art contrastive learning framework. In our ablations, we evaluate different formulations of the compositional constraints and composition functions. We evaluate the proposed approach for the downstream tasks of action detection in UCF-101, and video summarization in the SumMe dataset. We achieve significant improvements over the baseline, i.e., 3.9% and 6.3% relative improvements for UCF-101 and SumMe respectively, showcasing the importance of compositional video representations.

Joint Multimedia Event Extraction from Video and Article

Visual and textual modalities contribute complementary information about events described in multimedia documents. Videos contain rich dynamics and detailed unfoldings of events, while text describes more high-level and abstract concepts. However, existing event extraction methods either do not handle video or solely target video while ignoring other modalities. In contrast, we propose the first approach to jointly extract events from video and text articles. We introduce the new task of Video MultiMedia Event Extraction (Video M2E2) and propose two novel components to build the first system towards this task. First, we propose the first self-supervised multimodal event coreference model that can determine coreference between video events and text events without any manually annotated pairs. Second, we introduce the first multimodal transformer which extracts structured event information jointly from both videos and text documents. We also construct and will publicly release a new benchmark of video-article pairs, consisting of 860 video-article pairs with extensive annotations for evaluating methods on this task. Our experimental results demonstrate the effectiveness of our proposed method on our new benchmark dataset. We achieve 6.0% and 5.8% absolute F-score gain on multimodal event coreference resolution and multimedia event extraction.

Earlier Projects

Encoding Unertainity in Video Representation Learning

Can stochasticity of future in videos help learn better representations ? To answer this question, we relax the point-based contrastive loss to a n-shpere based contrastive loss. We build our experiments on top of Dense Predictive Coding, a self-supervised representation technique. Training on Kinetics-400 and fine-tuning on UCF-101, HMDB-51 help us understand that the later two datasets do not contain videos with diverse futures. Hence, we build a block toy dataset to control of stochasticity and evaluate the technique.

Causal Physical Reasoning

In this project, we take a step towards learning causal visual features. For this, we have created a simulation environment called Causal-PHYRE. For each task in the dataset, we control the outcome, cause and confounder. We start out by observing if a vanilla NN architecture detect the causal structure of tasks from simulated videos. Further, we simulate interventions in the environment to explicitly aid learning causal structure of underlying tasks. Experiments show that intervention-based learning strategy not only improves the detection of real cause rather than confounder. This concurres with assertion in Pearl Causal Hierarchy (PCH), which says that it is impossible to learn causal structure using just interventional data.

Optional Depth Pathway for Mask R-CNN

Instance segmentation is one of the most important perception tasks in computer vision. We present an approach to optionally use the depth given along an image to aid the performance on this task. We observe that depth information incorporated through Spatially-Adaptive (DE)normalization (SPADE) results in significant improvement on the task on NYUv2 dataset. Additionally, we observe that the use of ODM (Optional Depth Module) helps to prevent the degradation of performance even when the depth data is unavailable to the network.

Beyond Supervised Learning: A Computer Vision Perspective

Fully supervised deep learning-based methods have created a profound impact in various fields of computer science. Compared to classical methods, supervised deep learning-based techniques face scalability issues as they require huge amounts of labeled data and, more significantly, are unable to generalize to multiple domains and tasks. In recent years, a lot of research has been targeted towards addressing these issues within the deep learning community. Although there have been extensive surveys on learning paradigms such as semi-supervised and unsupervised learning, there are few timely reviews after the emergence of deep learning. In this paper, we provide an overview of the contemporary literature surrounding alternatives to fully supervised learning in the deep learning context. First, we summarize the relevant techniques that fall between the paradigm of supervised and unsupervised learning. Second, we take autonomous navigation as a running example to explain and compare different models. Finally, we highlight some shortcomings of current methods and suggest future directions.

Teaching

I have a keen interest in community learning. While at IIIT-H, I actively took initiatives to share what I learn. I totally believe in the following adage :

Bayesian Machine Learning

A short course on Bayesian Machine Learning covering EM, Variational Inference and Latent Variable Models in a succinct manner.

Linear Algebra

A short course on Linear Algebra covering matrix transformations and some form of matrix decompositions.