What does the model see?
Video ingestion determines which visual tokens reach a multimodal model: frame extraction rate, clip length, input resolution and compression. We measure how these preprocessing choices change its video understanding.
Research
One question, three parts
Our thesis is that frontier multimodal models can simulate the brain's response to video, replacing the costly fMRI datasets behind models like TRIBE v2. Testing it splits one question into three parts: do multimodal models process video like the human brain?
01
02
How does the brain respond?
Brain-encoding models such as Meta's TRIBE v2 predict fMRI responses to naturalistic video, audio and language. We test how well those predictions generalize to short-form, fast-cut video outside their training distribution.
03
Where do they line up?
We compare model representations with predicted cortical responses using encoding and representational similarity analyses, and report model-generated estimates, behavioral measurements and predicted neural responses as distinct signals.
Publications
Papers & reports
Our first write-ups are in progress. Papers will be listed here as they're released, and the blog will announce each one.