Research

We develop artificial intelligence tools for behavioral and neural data analysis, and conversely learn from the brain to solve challenging artificial intelligence problems.

Research Direction 1

Artificial Intelligence for Behavior Analysis

We strive to develop tools for the analysis of animal behavior. Behavior is a complex reflection of an animal's goals, state and character. Thus, accurately measuring behavior is crucial for advancing basic neuroscience, as well as the study of various neural and psychiatric disorders. However, measuring behavior (from video) is also a challenging computer vision and artificial intelligence problem.

Artificial Intelligence for Behavior Analysis - Figure 2

Artificial Intelligence for Behavior Analysis - Figure 3

Key Tools & Projects

DeepLabCut

Markerless pose estimation for animals

GitHub

DLC2action

Behavior segmentation from pose data

GitHub

LLaVAction

Multi-modal large language models for action understanding

Website

BehaveMAE

Masked autoencoders for hierarchical behavior

GitHub

WildCLIP

Vision-language models for wildlife

GitHub

Research Direction 2

Embodied AI & Musculoskeletal Control

Watching an expert athlete makes it obvious that brains have mastered the elegant control of our bodies — an astonishing feat given slow biological hardware and the sensory and motor latencies that constantly impede control. Understanding how the brain produces skilled movement is one of the central questions in neuroscience, and building agents that move as capably is one of the hardest open problems in AI. We work on both at once, using reinforcement learning, control theory, and curriculum learning to train biomechanically realistic, muscle-actuated models of the body.

Embodied AI & Musculoskeletal Control - Figure 2

Key Tools & Projects

DMAP

Distributed morphological attention for locomotion

GitHub

Lattice

Efficient exploration for motor control

arXiv

MyoChallenge 2022

Winning solution (NeurIPS 2022)

GitHub

MyoChallenge 2023

Winning solution (NeurIPS 2023)

GitHub

Research Direction 3

Task-Driven and Data-Driven Models of Proprioception as well as Sensorimotor Processing

We develop normative theories and models for sensorimotor transformations and learning. Work in the past decade has demonstrated that networks trained on object-recognition tasks provide excellent models for the visual system. Yet, for sensorimotor circuits this fruitful approach is less explored, perhaps due to the lack of datasets like ImageNet.

Task-Driven and Data-Driven Models of Proprioception as well as Sensorimotor Processing - Figure 2

Key Tools & Projects

proprioceptive illusions

Deep-learning models of the ascending proprioceptive pathway are subject to illusions

GitHub

task-driven primate

Task-driven neural network models predict neural dynamics of proprioception by Marin Vargas* and Bisi* et al.

GitHub

Our Mission

Our work strives to understand how the brain creates complex behavior. We develop tools for measuring behavior to achieve that goal, while ensuring they are broadly accessible to the community.

We make models and theories to elucidate how the brain gives rise to behavior with a specific focus on motor control and sensorimotor learning. Measuring behavior is key for assessing and constraining these models.