Best of Multimodal — 2025

1. 1
   Article

   ByteByteGo·41w

   How LLMs See Images, Audio, and More

   Modern AI systems process images, audio, and video by converting them into discrete tokens, similar to text processing. Images use patch embeddings (dividing into grid squares), vector quantization (learning visual codebooks), or contrastive embeddings. Audio employs neural codecs for quality preservation, ASR transcription for semantic content, or hierarchical approaches for multi-scale representation. Each tokenization method involves trade-offs between computational efficiency, information preservation, and semantic understanding, with the optimal choice depending on specific use cases and requirements.

   138
2. 2
   Article

   Salesforce Engineering·33w

   Building Real-Time Multimodal AI Pipelines

   Salesforce engineering team built real-time multimodal AI capabilities for Prompt Builder that process PDFs, images, and documents without pre-indexing. The system handles 50 million daily file uploads through a unified architecture serving both Data Cloud and non-Data Cloud customers. Key innovations include a real-time file processing pipeline with base64 conversion, a compatibility abstraction layer for multiple LLM providers (OpenAI, Gemini, Anthropic), and partial grounding validation that processes files independently rather than failing entire workflows. The solution unlocks file-based business data for AI agents, enabling use cases like automated document field extraction, insurance claim assessments, and case attachment summarization.

   37
3. 3
   Article

   Daily Dose of Data Science | Avi Chawla | Substack·43w

   Build a Multimodal Agentic RAG

   A comprehensive guide to building a multimodal agentic RAG system that processes both documents and audio files using speech input. The tutorial covers the complete workflow from data ingestion and audio transcription with AssemblyAI, to embedding storage in Milvus vector database, and orchestration with CrewAI Flows. The system allows users to query information using voice commands, with agents retrieving relevant context and generating cited responses. The implementation includes deployment using Beam for serverless containers and a Streamlit interface for user interaction.

   16
4. 4
   Article

   Daily Dose of Data Science | Avi Chawla | Substack·44w

   Build the Ultimate MCP Server for Multimodal AI

   A comprehensive guide to building an MCP (Model Context Protocol) server that enables multimodal AI capabilities across text, images, audio, and video. The tutorial demonstrates using Pixeltable as the multimodal AI infrastructure and CrewAI for orchestrating agent workflows. The system includes specialized agents for different modalities, a router agent for query classification, and a synthesis agent for response generation. The implementation supports RAG (Retrieval-Augmented Generation) operations across all media types through Docker-deployed MCP servers.

   14

   1
5. 5
   Article

   Hugging Face·1y

   Vision Language Models (Better, Faster, Stronger)

   This post reviews the developments in vision language models over the past year, highlighting new model architectures, specialized capabilities, and emerging paradigms. It covers trends such as any-to-any models, reasoning models, smaller yet capable models, and multimodal safety models, offering insights into how these innovations are shaping the future of AI.

   12