Week 1 Jan 20

LectureIntroduction

• Introduction to embodied learning

Week 2 Jan 27

LectureDeep Learning for Structured Outputs

TutorialHPC tutorial

• Deep learning for structured outputs
• Graphical models, energy-based models
• Autoregressive models
• Normalizing flows

Week 3 Feb 3

Lecture3D Vision, Mapping

TutorialEmbodied Simulators [Slides] [Code]

• Diffusion models
  • Probabilistic foundation
  • Applications in embodied learning
• 3D network designs
  • Bird’s eye view networks
  • Point cloud networks
  • Equivariance
  • Sensor fusion
  • Multi-task architecture
• Physical grounding
  • Stereo, self-supervised depth

Week 4 Feb 10

LectureSelf-Supervised Representation Learning and Object Discovery

TutorialVideo Learning [Slides]

• Physical grounding
  • Stereo, self-supervised depth
  • Optical flow
  • Unsupervised flow, depth and pose
• Mapping
  • Soft mapping
  • Registration
• Representation learning
  • DAE, MAE
  • Energy-based models
  • SSL, JEPA

Week 5 Feb 17

Legislative Monday

• No lecture

Week 6 Feb 24

LectureWorld Models

• Representation learning
  • Joint embedding models
• Object discovery
  • Pseudo-labels
  • Slot-based models
  • Complex-valued autoencoders
• World models
  • Trajectory prediction
  • Occupancy volume prediction
  • Video world models
  • Latent world models

Week 7 Mar 3

LectureContinual Learning, Memory, and Few-Shot Learning

TutorialSeeing → World Modeling [Slides]

• Continual learning

• Memory

• Few-shot learning

Week 8 Mar 10

Guest Lecture - Jorge Mendez-Mendez (Stony Brook University)

-

Week 9 Mar 17

Spring Break

• No lecture

Week 10 Mar 24

-

-

Week 11 Mar 31

-

-

Week 12 Apr 7

-

-

Week 13 Apr 14

-

-

Week 14 Apr 21

-

-

Week 15 Apr 28

-

-

Week 16 May 5

-

-