Frame extraction & stimulus alignment
Each clip is decoded, its frames extracted at a fixed temporal sampling rate and aligned with the audio track, so a multimodal model and a brain-encoding model receive an identical stimulus on a shared timeline.
Our thesis
Traditionally, understanding the human brain's neural response to video has required large neural datasets. They are hard to acquire because of device limitations, human‑subject testing regulations and the financial constraints faced by smaller research labs.
The recent state of the art shows how heavy that path is. Meta's TRIBE v2, a foundation model that predicts fMRI responses to video, audio and language, was built on a unified dataset of over 1,000 hours of fMRI from 720 subjects. Image‑reconstruction work such as MindEye2 depends on the Natural Scenes Dataset, which scanned 8 people in a 7‑Tesla MRI over dozens of sessions each. Models like these need scanner time, ethics approval and compute that few labs can afford.
We believe that as multimodal models from frontier labs like Anthropic and OpenAI grow larger in scale and consume a wider variety of data, they can be used to simulate these neural responses, without human testing and data collection.
The goal of BV Labs is to find out whether they can, and to enable further research into video understanding and ingestion.
Our approach
Representational comparison
We extract layer-wise activations from multimodal models and relate them to predicted fMRI responses using voxel-wise encoding models and representational similarity analysis, correcting for hemodynamic lag.
Divergence analysis
We localize where alignment breaks down by cortical region, by stimulus feature (motion, faces, scene cuts, on-screen text) and over time, to identify what video understanding models miss and what they over-weight.
