MOTION PROMPTING

Summary from the TLDR AI newsletter

Motion Prompting is a method that trains a video generation model with novel input types. Instead of just text or an image, it uses text, first image frame, and a tracking field of pixels. This then allows for novel control at inference time where it can provide a new pixel field (e.g., for an object moving in a new direction) and generate a video based on that. The method is extremely compelling, but not open sourced.