Ayush Tewari ayusht@mit.edu I am a postdoctoral researcher at MIT CSAIL with Bill Freeman, Josh Tenenbaum, and Vincent Sitzmann. I completed my Ph.D. at the Max Planck Institute for Informatics with Christian Theobalt, and I continue to be associated with the institute as a researcher. I develop methods that infer rich 3D structured representations of the visual world from images and videos, much like the mental models humans infer to interact with and navigate their surroundings. Check out some recent talks for an overview. I am on the faculty job market this year. [Google Scholar] [CV] [Publications] [Talks]

Publications

• Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision NeurIPS 2023 (Spotlight) Ayush Tewari*, Tianwei Yin*, George Cazenavette, Semon Rezchikov, Joshua B. Tenenbaum, Fredo Durand, William T. Freeman, and Vincent Sitzmann (* equal contribution) [project page] [paper] [code]

• FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow NeurIPS 2023 Cameron Smith, Yilun Du, Ayush Tewari, and Vincent Sitzmann [project page] [paper] [code]

• Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering SIGGRAPH Asia 2023 Linjie Lyu, Ayush Tewari, Marc Habermann, Shunsuke Saito, Michael Zollhoefer, Thomas Leimkuehler, and Christian Theobalt [project page] [paper]

• AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars SIGGRAPH Asia 2023 Mohit Mendiratta, Xingang Pan, Mohamed Elgharib, Kartik Teotia, Mallikarjun B R, Ayush Tewari, Vladislav Golyanik, Adam Kortylewski, and Christian Theobalt [project page] [paper]

• ModalNeRF: Neural Modal Analysis and Synthesis for Free-Viewpoint Navigation in Dynamically Vibrating Scenes EGSR, Computer Graphics Forum 2023 Automne Petitjean, Yohan Poirier-Ginter, Ayush Tewari, Guillaume Cordonnier, and George Drettakis [project page] [paper]

• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold SIGGRAPH 2023 Xingang Pan, Ayush Tewari, Thomas Leimkuehler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt [project page] [paper] [code]

• Learning to Render Novel Views from Wide-Baseline Stereo Pairs CVPR 2023 Yilun Du, Cameron Smith, Ayush Tewari†, and Vincent Sitzmann† († equal advising) [project page] [paper] [code]

• ConceptFusion: Open-set Multimodal 3D Mapping RSS 2023 Krishna Murthy Jatavallabhula, ..., Ayush Tewari, et. al [project page] [paper]

• Neural Groundplans: Persistent Neural Scene Representations from a Single Image ICLR 2023 Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Ambrus, Adrien Gaidon, William T. Freeman, Fredo Durand, Joshua B. Tenenbaum, and Vincent Sitzmann [project page] [paper]

• Exploring Perceptual Straightness in Learned Visual Representations ICLR 2023 Anne Harrington, Vasha DuTell, Ayush Tewari, Mark Hamilton, Simon Stent, Ruth Rosenholtz, and William T. Freeman [paper]

• Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination ECCV 2022 (Oral Presentation) Linjie Lyu, Ayush Tewari, Thomas Leimkuehler, Marc Habermann, and Christian Theobalt [project page] [paper]

• GAN2X: Non-Lambertian Inverse Rendering of Image GANs 3DV 2022 Xingang Pan, Ayush Tewari, Lingjie Liu, and Christian Theobalt [project page] [paper]

• VoRF: Volumetric Relightable Faces BMVC 2022 (Oral Presentation, Best Paper Honorable Mention) Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, and Mohamed Elgharib [project page] [paper]

• Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images CVPR 2022 Ayush Tewari, Mallikarjun B R, Xingang Pan, Ohad Fried, Maneesh Agrawala, and Christian Theobalt [project page] [paper]

• Advances in Neural Rendering Eurographics State-of-the-Art Report 2022 Ayush Tewari*, Justus Thies*, Ben Mildenhall*, Pratul Srinivasan*, Edith Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stepehen Lombardi, Tomas Simon, Christian Theobalt, Matthias Niessner, Jonathan T. Barron, Gordon Wetzstein, Michael Zollhoefer, and Vladislav Golyanik (* equal contribution) [paper]

• StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN BMVC 2021 (Oral Presentation) Gereon Fox, Ayush Tewari, Mohamed Elgharib, and Christian Theobalt [project page] [paper]

• Efficient and Differentiable Shadow Computation for Inverse Problems ICCV 2021 Linjie Lyu, Marc Habermann, Lingjie Liu, Mallikarjun B R, Ayush Tewari, and Christian Theobalt [project page] [paper]

• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video ICCV 2021 Edith Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhoefer, Christoph Lassner, and Christian Theobalt [project page] [paper] [code]

• PhotoApp: Photorealistic Appearance Editing of Head Portraits SIGGRAPH 2021 Mallikarjun B R, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, and Christian Theobalt [project page] [paper]

• i3DMM: Deep Implicit 3D Morphable Model of Human Heads CVPR 2021 (Oral Presentation) Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt [project page] [paper] [code]

• Monocular Reconstruction of Neural Face Reflectance Fields CVPR 2021 Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, and Christian Theobalt [project page] [paper]

• Learning Complete 3D Morphable Face Models from Images and Videos CVPR 2021 Mallikarjun B R, Ayush Tewari, Hans-Peter Seidel, Mohamed Elgharib, and Christian Theobalt [project page] [paper]

• Monocular Real-time Full Body Capture with Inter-part Correlations CVPR 2021 Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, and Feng Xu [project page] [paper]

• 3D Morphable Face Models - Past, Present and Future ACM Transactions on Graphics 2021 Bernhard Egger, Will Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter [paper]

• Egocentric Videoconferencing SIGGRAPH Asia 2020 Mohamed Elgharib*, Mohit Mendiratta*, Justus Thies, Matthias Niessner, Hans-Peter Seidel, Ayush Tewari, Vladislav Golyanik, and Christian Theobalt (* equal contribution) [project page] [paper]

• PIE: Portrait Image Embedding for Semantic Control SIGGRAPH Asia 2020 Ayush Tewari, Mohamed Elgharib, Mallikarjun B R, Florian Bernard, Hans-Peter Seidel, Patrick Perez, Michael Zollhoefer, and Christian Theobalt [project page] [paper]

• PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations ECCV 2020 Edith Tretschk, Ayush Tewari, Michael Zollhoefer, Vladislav Golyanik, and Christian Theobalt [project page] [paper]

• Neural Voice Puppetry: Audio-driven Facial Reenactment ECCV 2020 Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Niessner [project page] [paper] [code]

• State of the Art on Neural Rendering Eurographics State-of-the-Art Report 2020 Ayush Tewari*, Ohad Fried*, Justus Thies*, Vincent Sitzmann*, Stepehen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Niessner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Schectman, Dan B Goldman, and Michael Zollhoefer (* equal contribution) [paper]

• StyleRig: Rigging StyleGAN for 3D Control over Portrait Images CVPR 2020 (Oral Presentation) Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Perez, Michael Zollhoefer, and Christian Theobalt [project page] [paper]

• Text-based Editing of Talking-head Video SIGGRAPH 2019 Ohad Fried, Ayush Tewari, Michael Zollhoefer, Adam Finkelstein, Eli Schectman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala [project page] [paper]

• FML: Face Model Learning from Videos CVPR 2019 (Oral Presentation) Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Perez, Michael Zollhoefer, and Christian Theobalt [project page] [paper]

• High-Fidelity Monocular Face Reconstruction based on an Unsupervised Model-based Face Autoencoder TPAMI 2018, special issue on The Best of ICCV 2017 Ayush Tewari, Michael Zollhoefer, Florian Bernard, Pablo Garrido, Hyeongwoo Kim, Patrick Perez, and Christian Theobalt [project page] [paper]

• A Hybrid Model for Identity Obfuscation by Face Replacement ECCV 2018 Qianru Sun, Ayush Tewari, Weipeng Xu, Mario Fritz, Christian Theobalt, and Bernt Schiele [project page] [paper]

• Deep Video Portraits SIGGRAPH 2018 Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Perez, Christian Richardt, Michael Zollhoefer, and Christian Theobalt [project page] [paper]

• Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz CVPR 2018 (Oral Presentation) Ayush Tewari, Michael Zollhoefer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Perez, and Christian Theobalt [project page] [paper]

• InverseFaceNet: Deep Single-Shot Inverse Face Rendering From A Single Image CVPR 2018 Hyeongwoo Kim, Michael Zollhoefer, Ayush Tewari, Justus Thies, Christian Richardt, and Christian Theobalt [project page] [paper]

• MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction ICCV 2017 (Oral Presentation) Ayush Tewari, Michael Zollhoefer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt [project page] [paper]

Talks

• Structured Generative Models, John Hopkins University, CMU, Meta Reality Labs Research, Princeton, 2023
  [slides]
  
• Teaching AI to See the 3D World, CSAIL + Imagination in Action: AI Frontiers & Implications, 2023
  [video]
  
• Finding 3D Structure in Unstructured 2D Data, Rank Prize Seminar, Oxford University, Adobe, 2022, Princeton University, 2023
  [slides]
  
• Learning 3D Generative Models from 2D Data, Dagstuhl Seminar, 2022
  [slides]
  
• Synthesis of Portrait Images with 3D Control, CVPR NTIRE workshop 2021
  [slides]
  
• Self-Supervised 3D Digitization of Faces, MIT vision and graphics seminar 2021
  [video]
  
• GANs with 3D Control, SIGGRAPH course 2021
  [video]
  
• PIE: Portrait Image Embedding for Semantic Control, SIGGRAPH Asia 2020
  [video]
  
• StyleRig: Rigging StyleGAN for 3D Control over Portrait Images, CVPR 2020
  [video]
  
• Neural Rendering Fundamentals, CVPR 2020 tutorial
  [video]
  
• FML: Face Model Learning from Videos, CVPR 2019
  [video]
  
• Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz, CVPR 2018
  [video]
  
• MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction, ICCV 2017
  [video]