Science & Technology

Building Multimodal Models

📚 Main topics

Three approaches to multimodal modelsThe stream compares Chameleon from Meta, Hugging Face’s Idefics 2, and Google’s Mirasol 3B to discuss different ways of combining text, images, audio, and video. 2:46
Idefics 2 and the open-source approachHugging Face’s model connects a pretrained vision encoder to a pretrained language model with a learned projection layer, making it more practical to build with limited compute. 20:26
Chameleon’s early fusion designMeta trains a model from scratch on mixed text-and-image data, converting both modalities into tokens and allowing the model to generate interleaved text and images. 6:55
Mirasol’s video and audio processingGoogle’s model encodes audio and video in time-aligned chunks, uses a shared vision backbone for video and spectrograms, and produces text. 42:35
Memory in multimodal systemsMirasol’s Token Turing Machine uses Transformer components and external memory to process and retain information, a design the speaker relates to recurrent models. 59:06

✨ Key takeaways

Training choices depend on computeReusing pretrained components makes multimodal modeling more accessible, while training an end-to-end model from scratch requires enormous compute and infrastructure. 35:20
Mixed-modality training may helpThe Chameleon results suggest that training on interleaved text and image data can improve performance on text-only benchmarks as well as multimodal tasks. 11:29
Image tokenization has limitationsCompressing images into discrete tokens can lose detail, especially small text, making OCR a persistent challenge. 19:23
Different architectures suit different constraintsCross-attention can help information flow between modalities, but it can also increase compute and complexity; the best choice depends on the model and training setup. 47:44
Multimodal outputs add practical complexityA model generating both text and images must handle modality-specific tokens, image boundaries, and complete image generation. 53:57

🧠 Lessons learned

Tokenization mattersText and images need to be converted into sequences the model can process, and choices such as vocabulary or codebook size affect what information the model can represent. 16:14
Training stages need careful mixingWhen shifting toward higher-quality data, keeping some earlier training data in the mixture can help reduce catastrophic forgetting. 28:40
More modalities bring new training challengesModalities can compete during training, so methods such as query-key normalization and dropout may be needed to stabilize learning. 32:15
Avoid overreading benchmark claimsThe speaker recommends paying attention to what a model can actually do and how it behaves, rather than relying solely on benchmark scores. 1:16:19
Simplicity is a useful guiding principleThe speaker argues that, rather than continually adding specialized modules, progress may come from simpler end-to-end models trained on more data for longer. 1:29:21

🏁 Conclusion/next steps

Toward end-to-end multimodalityThe speaker expects future systems to combine audio, video, and text more directly, with models capable of both consuming and generating multiple modalities. 1:17:20
Memory and inference remain open challengesStateful multimodal models will need ways to store and retrieve information efficiently, especially across long periods or separate interactions. 1:11:32
Small-scale research can still be worthwhileAlthough smaller projects may not match large-company training runs, the speaker encourages researchers to pursue work that is meaningful and fulfilling to them. 1:43:58

Transcript excerpt

0:00 testing on YouTube testing on YouTube test test test test test test test test test testing on YouTube testing on YouTube testing on YouTube come on

0:33 why this not why is this not streaming all right there we go audio is working to nice all right last one make sure this works on x testing on X testing on

1:04 X how's it going Christopher how's it going Sarah how's it going acram how's it going exploring Lord Sheba Dark Matter Unice how's it going everybody pruk dan acromag again all right should be live on X any second

1:39 no how's it going day all right let's confirm that the audio works on X and then we'll get started broadcast is not available broadcast is not available get your servers figured out X hire some Netflix people all right let's confirm the audio works okay perfect it's working or YouTube people I don't know what I guess YouTube and Netflix are probably the best video streaming

2:12 sites all right I'm going take a sip of water and then we'll start this [Music] all right welcome everybody to another hoost stream we're going to be going into the world of multimodal models today so building mul multimodal models

🔒 The full, searchable transcript is available with Pro.

Suggestions