All Publications

CVPR 2024      [PDF] [Project page]

An LLM agent for visual program synthesis trainable with reinforced self-training using just weak supervision from vision-language tasks.

CVPR 2024      [PDF]

A Lidar-enhanced neural radiance field to transform drive videos into photorealistic sensor simulation testbeds.

CVPR 2024      [PDF] [Project page]

Transfer photorealistic, high-fidelity and geometry-aware textures from 3-5 images to arbitrary 3D meshes.

CVPR 2024      [PDF]

A VLM and LLM-based framework for issue-finding, auto-labeling, continual training and automated verification for visual perception in autonomous driving.

CVPR 2024      [PDF] [Project page]

Scale neural volume rendering to high resolution by rendering every pixel to ensure that "what you see in 2D, is what you get in 3D".

SIGGRAPH Asia 2023      [PDF] [Project page] [Code]

A general Monte Carlo volume rendering algorithm to accelerate any NeRF representation using importance sampling based on ray density distributions.

ToG 2023      [PDF]

A physically-motivated deep network that predicts spatiotemporally consistent lighting given a single LDR video of an indoor scene.

NeurIPS 2023      [PDF] [Project page] [Code]

Question decomposition that allows multi-billion scale vision-language models to approach reasoning-heavy visual question-answering as a two-step rather than a single-step problem.

CVPR 2023      [PDF]

Learn dense 3D object shapes for autonomous driving, using differentiable rendering with instance masks and 3D boxes, along with category-level priors.

CVPR 2023      [PDF] [Project page] [Code]

A large-scale dataset and study on the robustness of computer vision algorithms and large pre-trained models across geographical biases.

WACV 2023 [Best Paper Finalist]      [PDF] [Project page] [Code]

A flow-free and single-shot prediction approach for video frame interpolation, allowing efficient ultra-slow motion effects using just 3D convolutions.

ICCV 2023 (Oral)      [PDF] [Project page] [Code]

Joint estimation of emitters and materials in inverse rendering, to enable high-quality AR applications like scene relighting and object insertion.

ICCV 2023      [PDF] [Code]

Pseudo-label question-answers in unlabeled images, for fine-tuning an existing large vision-language model on visual question answering in a data-scarce target dataset.

ICCV 2023      [PDF] [Project page]

Inverse rendering through variational optimization of image functionals that allows for discrete topology changes by nucleating hole or phase changes, with applications to multiview 3D reconstruction and text-to-image generation.

SIGGRAPH 2023      [PDF] [Project page]

A real-time 3D GAN with a ViT encoder to infer and render a photorealistic 3D representation from a single unposed image such as a face portrait.

ECCV 2022 (Oral)      [PDF] [Project page] [Code] [Video]

Estimation of visible and invisible light sources from a single image of an indoor scene, to enable scene relighting and object insertion with full global effects.

ECCV 2022 (Oral) [Best Paper Honorable Mention]      [PDF] [Project page] [Code] [Video]

A level-set theory bridge between explicit and implicit shape representations that allows user-defined editing of neural implicit geometry.

ECCV 2022      [PDF] [Project page] [Code]

Unsupervised domain adaptation that scales to a large number of categories using a memory bank.

ECCV 2022      [PDF]

Semantic segmentation training across multiple datasets with differing label spaces for better generalization to unseen domains.

ECCV 2022      [PDF]

Object detection for unseen categories using pseudo-labels generated with large pre-trained vision-language models.

ECCV 2022      [PDF]

Single-stream architecture for aligning vision and language representations at instance, patch and semantic levels..

ECCV 2022      [PDF]

Photorealistic virtual environments with high-quality multi-modal ground truth for geometry, material and semantics in road scenes.

CVPR 2022 (Oral)      [PDF] [Project page] [Code] [Video]

A vision transformer architecture for inverse rendering in indoor scenes that allows learning long-range global interactions.

CVPR 2022      [PDF] [Project page] [Code] [Video]

A framework that uses a set of images of an indoor scene to create a photorealistic digital twin with high-quality material and lighting.

CVPR 2022      [PDF]

Single-image occlusion-reasoned road layout estimation in both perspective and parametric top-view space.

CVPR 2022      [PDF]

A continual learning method that allows generalization to new tasks in new domains with limited labels.

CVPR 2022      [PDF]

Meta-learning to address biases due to long-tailed data and factors of variation such as ethnicity, head pose, occlusion and blur.

CVPR 2022 (Oral)      [PDF]

Multi-task learning with a dynamic architecture that satisfies resource constraints and user preferences.

CVPR 2022      [PDF]

A clustering objective enforced in a transformed feature space for domain adaptation across segmentation datasets with completely different categories.

ICCV 2021      [PDF] [Project page] [Code]

Local functional representations that encode signals while achieving both high accuracy and generalization ability.

ICCV 2021      [PDF]

Unsupervised domain adaptation for video representations in action recognition by aligning features across color and optical flow modalities.

CVPR 2021 (Oral)      [PDF] [Project page] [Code] [Video]

An open source dataset of indoor scenes with high-quality tools and ground truth for shape, material and lighting, for augmented reality and robotics applications.

Sriram Narayanan, Ramin Moslemi, Francesco Pittaluga, Buyu Liu, Manmohan Chandraker

A multi-choice learning objective for diverse future trajectory prediction and a lane anchor network for 3D scene-consistent outputs.

Astuti Sharma, Tarun Kalluri, Manmohan Chandraker

An instance-affinity based criterion with a multi-sample contrastive loss for improved domain alignment.

Bingbing Zhuang, Manmohan Chandraker

A probabilistic fusion that learns to account for relative uncertainties in pose estimation by traditional geometric methods and deep neural networks, to achieve both better generalization and handling of ambiguities.

Road scene parsing in bird-eye view combining the relative benefits of ground-based proximate images with aerial remote sensing inputs.

Neural ODEs to generate 3D meshes with guaranteed manifoldness, enabling physically meaningful applications like rendering, simulations and 3D printing.

A physically-motivated network for joint shape and material estimation, as well as relighting under novel illumination conditions, using a single image captured by a mobile phone camera.

Constant-time, scene consistent and diverse trajectory prediction regardless of number of agents, along with a dynamics simulator that diversifies top-views of real scenes.

Recover object height and camera parameters from a single unconstrained image, through a network that imbibes weakly supervised geometric constraints through bounding boxes and the horizon.

A clustering method to improve face recognition using unlabeled data, exploiting extreme value theory to account for overlapping identies between labeled and unlabeled data.

A two-layer convolutional LSTM that uses insights from keyframe-based methods to learn visual odometry from very long monocular sequences, in contrast to prior methods that are limited to short snippets.

A self-improving framework for monocular SLAM that iterates between unsupervised depth estimation and robust geometric SLAM to achieve state-of-the-art on KITTI and TUM sequences.

Domain adaptation that uses weak image-level labels to achieve category-wise alignment despite distribution shifts in dense structured prediction problems like semantic segmentation.

A method for training a single object detector across multiple datasets, despite overlaps or mismatches between their label spaces.

A physically-motivated network that models refractions and reflections to recover high-quality 3D geometry for complex transparent shapes using as few as 5-12 natural images.

A physically-motivated network that recovers shape, complex material and spatially varying lighting from a single mobile phone image to enable photorealistic indoor AR applications.

A novel planar representation for indoor 3D scene understanding that reasons about occlusions, with new metrics and a new dataset.

A universal representation for face recognition that achieves robustness and accuracy across several factors of variation such as blur, resolution and occlusion.

Predict top-view layout of a complex 3D scene using a video sequence as input, with spatial consistency among objects and scene elements, as well as temporal coherence.

Making differential privacy practical for computer vision by reducing more than 90% privacy cost compared to prior methods, while achieving better accuracy.

An active learning framework by integrating domain adversarial learning and importance sampling for continuous semi-supervised domain adaptation, applied to object recognition and detection.

Video deblurring with an internal attention module to select optimal temporal scales for restoring the sharp center frame and an external attention module to aggregate such estimates across different blur levels.

Action recognition on drone videos without using labeled data, through adversarial domain adaptation to labeled Internet video datasets.

Learning deep features that protect against model inversion attacks by an adversary with access to large-scale public data, with application to facial attribute recognition while protecting identity.

Semantic segmentation for automatic microscopy image analysis to quantify oncogenes located on extrachromosomal DNA.

Learning fair representations that perform well across both data-rich and data-poor domains, applied to semantic segmentation that minimizes annotation and deployment costs.

Unsupervised domain adaptation for semantic segmentation by learning discriminative feature representations of patches in the source domain that correspond to modes of the output space distribution.

A parametric representation for 3D scene understanding of complex road scenes that is intuitive for human visualization and interpretable for higher-level decision making.

Mitigating bias against under-represented categories in face recognition by augmenting the feature space of under-represented subjects using the distribution of features for subjects that have sufficiently diverse samples.

Theoretical limits on SFM with a rolling-shutter camera and leveraging data-driven priors through a network that learns camera motion and scene stucture to undistort a single rolling shutter image.

Unsupervised domain adaptation that combines insights from semi-supervised learning for feature-level adaptation and 3D geometry-guided image synthesis for pixel-level adaptation.

A reinforcement learning-based method for automatically adjusting the parameters of any non-differentiable simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data.

Making face recognition fair across ethnicities through unsupervised adaptation across domains with non-overlapping label spaces, retaining identification power on all ethnicities while keeping representations for all identities well-separated.

Single-image depth map and normal estimation for transparent shapes using a network trained on a new synthetic dataset.

Taking a step towards self-driving on Indian roads through a novel large-scale dataset that provides instance segmentation and object detection labels.

A memory-based online video representation that achieves efficiency and accuracy through spatiotemporal warping to compensate for motions, while enabling prediction of feature representations in future frames.

A differential rendering layer with global illumination to train a network that recovers shape and complex, spatially-varying material using a single image acquired with a mobile phone camera.

A differentiable rendering layer that models image formation with a complex spatially-varying BRDF to recover high-quality material information using a single mobile phone image.

Predict occluded portions of the scene layout by looking around foreground objects like cars or pedestrians, with learned priors and rules about typical road layouts from simulated or, if available, map data.

Metric learning and deep supervision to leverage feature hierarchies in a deep convolutional network for 2D and 3D geometric matching.

Unsupervised domain adaptation for semantic segmentation that aligns spatial similarities across domains through a structured output space.

Learning appearance and geometric priors for single-image undistortion of an image observed through a dynamic refractive medium such as water waves.

Fast and accurate object detection network through knowledge distillation that transfers insights to a compact student model from a higher-capacity teacher model.

Video face recognition that utilizes large-scale unlabeled video data for adversarial domain alignment while transferring discriminative knowledge from large-scale labeled still images.

A GAN that utilizes 3D geometric priors to frontalize a profile view of a face, achieving high quality as well as maintaining identity.

Metric learning to explicitly disentangle identity and pose, by demanding alignment between feature reconstructions through various combinations of identity and pose features.

Weakly supervised 3D reconstruction using silhouettes through a raytrace pooling layer that enables perspective projection and backpropagation, along with an adversarial constraint.

A variational energy minimization framework for robust recovery of shape in multiview stereo with unknown BRDF, with applications to light field cameras.

Deep supervision to sequentially infer intermediate concepts associated with the final task allows better generalization, demonstrated with an application to 3D semantic parsing from 2D images.

Predicting future trajectories in complex 3D scenes by accounting for the multimodaility of future behaviors, scene semantics and interactions among objects.

Learning differentiable cost functions for multi-target tracking by association, with a bilevel optimization to minimize a loss defined on the solution of a liner program.

The PRW dataset, methods and metrics for jointly performing person detection and re-identification in videos.

Deep metric learning to learn a feature space for geometric and semantic correspondence, with novel architectures such as a convolutional spatial transformer to handle local variations.

Landmark localization for non-rigid objects by incorporating geometric insights such as shape bases and thin-plate spline deformations in a deep convolutional network.

A dataset of light field images and novel 4D convolutional architectures to improve material recognition by exploiting multiple sub-aperture views and view-dependent reflectance effects.

Single-view reconstruction of non-rigid objects like birds without using part annotations, through a deformation-aware synthetic data generation and spatial priors in a deep network.

Shape and spatially-varying material recovery from a single light field image of an object, using theoretical invariants from differential stereo with general reflectance.

Interpretable occlusion reasoning for 3D scene understanding that assigns points to objects by modeling reflection or transmission probabilities for the corresponding camera ray.

Scalable framework for monocular 3D scene understanding with a hierarchical model that captures co-occurence and mutual exclusion relationships while incorporating both low-level trajectory features and high-level scene features.

Theoretical inavriants that eliminate BRDF from a system of differential stereo equations to yield PDE constraints that delineate the extent of shape recovery.

Robust monocular SFM in road scenes through novel designs for long-term feature tracking and scale drift correction.

Single-camera 3D localization of objects in traffic scenes by combining cues from SFM point tracks and object detection bounding boxes.

Differential invariants under object or camera motion that constrain shape recovery or material estimation under various conditions on camera, material or lighting.

Specification of the extent of shape recovery through differential stereo invariants obtained using images of an object with unknown material observed under a small camera motion.

Scale drift correction in monocular SFM using cues such as sparse features, dense stereo and object bounding boxes, by relating observation covariances of cues to error behaviors of their underlying variables.

Theoretical analysis of shape recovery using differential flow invariants obtained under small motions of an object relative to its environment.

Semantic 3D reconstruction using shape priors consisting of a mean shape for the commonality of shapes across a category and weighted anchor points to encode similarities between instances in the form of appearance and spatial consistency.

The first demonstration of practical monocular visual odometry in extended traffic scenes, through novel multithreaded design for robust feature tracking and scale drift correction.

A photometric flow relation that specifies theoretical extent of shape recovery possible with differential motion of a light source.

A semiparametric regression for single-image material estimation that achieves better generalization and interpretability, enabling applications such as relighting and material editing.

Differential invariants under light source motion that determine the extent of shape recovery and prior information needed for it, using images of an object with unknown material.

Fast computational methods for light transport inversion through a duality with forward rendering that allows analogues to radiosity, Monte Carlo and wavelet-based methods.

Tight convex relations in a branch and bound framework for globally optimal estimation of the plane at infinity with chirality constraints and dual image of the absolute conic with semidefinite constraints, allowing metric upgrade of a projective reconsturction.

Establishes light transport inversion as the dual problem of forward rendering, enabling efficient new methods for projector radiometric compensation and separation of bounces of global illumination.

A fast solution for structure and motion from line correspondences through an overdetermined polynomial system, used in a robust RANSAC framework for line-based SFM.

Globally optimal solutions to bilinear problems through convex relations in a branch and bound framework, demonstrated for face 3D morphable model fitting and non-rigid SFM.

Insights from fractional programming and the theory of convex underestimators for globally optimal solutions in multiview geometry under the standard L2-norm of reprojection errors and the L1-norm which is less sensitive to outliers.

Globally optimal metric upgrade of a projective reconstruction, through efficient convex relaxations for estimating the plane at infinity and camera intrinsic parameters.

Enable multi-touch interactions on an arbitrary flat surface using a pair of cameras mounted above the surface, by robustly identifying fingertips and precisely detecting touch.

An algorithm for performing Lambertian photometric stereo in the presence of shadows. It reduces the low frequency bias inherent to the normal integration process and ensures that the recovered surface is consistent with the shadowing configuration.

An autocalibration algorithm for upgrading a projective reconstruction to a metric reconstruction by estimating the absolute dual quadric.

A practical method for finding the provably globally optimal solution to numerous problems in projective geometry including multiview triangulation, camera resectioning and homography estimation.

For general nonconvex surfaces, interreflections completely resolve the GBR ambiguity. The full Euclidean geometry can be recovered from uncalibrated photometric stereo for which the light source directions and strengths are unknown.

A new, fully mobile, purely vision-based pose tracking system that works indoors in a prepared room, using artificial landmarks.

A multistage approach to gray-level corner detection that detects corners as the intersection points of the involved edges only by using a small neighborhood of the estimated corner position.