selected research (more)

I’m interested in self-supervised learning, representation learning, curiosity-based exploration, and leveraging internet-scale models and data. I am keen to draw inspiration from intelligence in humans and nature—especially as a goal-post rather than a blueprint. My long-term goal is to develop intelligent agents that can generalize and continually adapt as robustly and efficiently as humans do, allowing them to be safely deployed in the real world.

publications

2024

1. arXiv

   V-IRL: Grounding Virtual Intelligence in Real Life

   Jihan Yang ,  Runyu Ding ,  Ellis Brown ,  Xiaojuan Qi ,  and  Saining Xie

   arXiv preprint arXiv:2402.03310, 2024

   Abs arXiv Bib PDF Code Website

   There is a sensory gulf between the Earth that humans inhabit and the digital realms in which modern AI agents are created. To develop AI agents that can sense, think, and act as flexibly as humans in real-world settings, it is imperative to bridge the realism gap between the digital and physical worlds. How can we embody agents in an environment as rich and diverse as the one we inhabit, without the constraints imposed by real hardware and control? Towards this end, we introduce V-IRL: a platform that enables agents to scalably interact with the real world in a virtual yet realistic environment. Our platform serves as a playground for developing agents that can accomplish various practical tasks and as a vast testbed for measuring progress in capabilities spanning perception, decision-making, and interaction with real-world data across the entire globe.

   @article{yang2024virl,
     author = {Yang, Jihan and Ding, Runyu and Brown, Ellis and Qi, Xiaojuan and Xie, Saining},
     title = {V-IRL: Grounding Virtual Intelligence in Real Life},
     journal = {arXiv preprint arXiv:2402.03310},
     year = {2024},
   }

2023

1. 

   Your Diffusion Model is Secretly a Zero-Shot Classifier

   Alexander C. Li ,  Mihir Prabhudesai ,  Shivam Duggal ,  Ellis Brown ,  and  Deepak Pathak

   In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023

   Abs arXiv Bib PDF Website

   The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. These models can generate realistic images for a staggering variety of prompts and exhibit impressive compositional generalization abilities. Almost all use cases thus far have solely focused on sampling; however, diffusion models can also provide conditional density estimates, which are useful for tasks beyond image generation.
   
   In this paper, we show that the density estimates from large-scale text-to-image diffusion models like Stable Diffusion can be leveraged to perform zero-shot classification without any additional training. Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models. Although a gap remains between generative and discriminative approaches on zero-shot recognition tasks, our diffusion-based approach has significantly stronger multimodal compositional reasoning ability than competing discriminative approaches.
   
   Finally, we use Diffusion Classifier to extract standard classifiers from class-conditional diffusion models trained on ImageNet. Our models achieve strong classification performance using only weak augmentations and exhibit qualitatively better "effective robustness" to distribution shift. Overall, our results are a step toward using generative over discriminative models for downstream tasks.

   @inproceedings{li2023diffusion,
     title = {Your Diffusion Model is Secretly a Zero-Shot Classifier},
     author = {Li, Alexander C. and Prabhudesai, Mihir and Duggal, Shivam and Brown, Ellis and Pathak, Deepak},
     year = {2023},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
     pages = {2206-2217},
   }

2. 

   Internet Explorer: Targeted Representation Learning on the Open Web

   Alexander C. Li* ,  Ellis Brown* ,  Alexei A. Efros ,  and  Deepak Pathak

   In International Conference on Machine Learning , 2023

   Abs arXiv Bib PDF Website

   Vision models heavily rely on fine-tuning general-purpose models pre-trained on large, static datasets. These general-purpose models only understand knowledge within their pre-training datasets, which are tiny, out-of-date snapshots of the Internet—where billions of images are uploaded each day. We suggest an alternate approach: rather than hoping our static datasets transfer to our desired tasks after large-scale pre-training, we propose dynamically utilizing the Internet to quickly train a small-scale model that does extremely well on the task at hand. Our approach, called Internet Explorer, explores the web in a self-supervised manner to progressively find relevant examples that improve performance on a desired target dataset. It cycles between searching for images on the Internet with text queries, self-supervised training on downloaded images, determining which images were useful, and prioritizing what to search for next. We evaluate Internet Explorer across several datasets and show that it outperforms or matches CLIP oracle performance by using just a single GPU desktop to actively query the Internet for 30–40 hours.

   @inproceedings{li2023internet,
     title = {Internet Explorer: Targeted Representation Learning on the Open Web},
     author = {Li*, Alexander C. and Brown*, Ellis and Efros, Alexei A. and Pathak, Deepak},
     year = {2023},
     booktitle = {International Conference on Machine Learning},
     organization = {PMLR},
   }

2018

1. 

   An Architecture for Spatiotemporal Template-Based Search

   Ellis Brown ,  Soobeen Park ,  Noel Wardord ,  Adriane Seiffert ,  Kazuhiko Kawamura ,  Joseph Lappin ,  and  Maithilee Kunda

   Advances in Cognitive Systems, 2018

   Abs Bib PDF

   Visual search for a spatiotemporal target occurs frequently in human experience—from military or aviation staff monitoring complex displays of multiple moving objects to daycare teachers monitoring a group of children on a playground for risky behaviors. In spatiotemporal search, unlike more traditional visual search tasks, the target cannot be identified from a single frame of visual experience; as the target is a spatiotemporal pattern that unfolds over time, detection of the target must also integrate information over time. We propose a new computational cognitive architecture used to model and understand human visual attention in the specific context of visual search for a spatiotemporal target. Results from a previous human participant study found that humans show interesting attentional capacity limitations in this type of search task. Our architecture, called the SpatioTemporal Template-based Search (STTS) architecture, solves the same search task from the study using a wide variety of parameterized mod els that each represent a different cognitive theory of visual attention from the psychological literature. We present results from initial computational experiments using STTS as a first step towards understanding the computational nature of attentional bottlenecks in this type of search task, and we discuss how continued STTS experiments will help determine which theoretical models best explain the capacity limitations shown by humans. We expect that results from this research will help refine the design of visual information displays to help human operators perform difficult, real-world monitoring tasks.

   @article{brown2018stts,
     title = {An Architecture for Spatiotemporal Template-Based Search},
     author = {Brown, Ellis and Park, Soobeen and Wardord, Noel and Seiffert, Adriane and Kawamura, Kazuhiko and Lappin, Joseph and Kunda, Maithilee},
     year = {2018},
     journal = {Advances in Cognitive Systems},
     volume = {6},
     pages = {101--118},
   }

code