For much of AI’s recent history, most tools were built around a single type of input, most commonly text. Today, multimodal systems capable of processing images, video, and language within a unified framework have become widely available – yet the challenge of genuine cross-modal reasoning remains far from solved.
On the surface, it seems like a natural progression. If large language models (LLMs) can generate coherent text and computer vision systems can reliably identify objects, then combining the two should create a more complete form of machine intelligence. Right?
Not exactly. In reality, multimodal AI is much more complex.
The challenges aren’t perception or language generation in isolation, but the integration of the two into systems that can reason reliably about what they see.
From recognition to reasoning
In recent years, AI has become good at recognizing what it sees, like spotting a tumor in a medical scan or identifying key details in a document. Meanwhile, LLMs have grown capable of explaining, summarizing, and following complex instructions with impressive fidelity.
Yet combining these two strengths doesn’t automatically deliver the results you might expect. A model might easily pinpoint every object in a given image, but entirely miss how those objects relate to one another. And it might still produce an explanation that sounds convincing but doesn’t align with the visual evidence, or lose track of context entirely, particularly in fast-changing scenes like video.
Researchers have begun paying closer attention to this. New multimodal-specific architectures combine visual encoders and language backbones to process text, images and video within a shared framework.
Ultimately, the ability to ‘see’ something clearly isn’t the same as knowing what it means. It’s true for humans and even truer for AI.
Why multimodal reasoning breaks down
At the core of the issue is alignment. Multimodal systems must align three distinct elements:
- Perception: accurately extracting features from images or video
- Representation: mapping those features into a form compatible with language models
- Reasoning: generating outputs that are logically consistent with both the input and the task
Failures can occur at any of these stages. Visual encoders could miss subtle but critical details, or connectors that translate visual information into language tokens might lose context. Or, if a language model is trained primarily on text, it may default to learned patterns rather than grounding responses in the actual visual input.
This misalignment leads to multimodal hallucination, which is, of course, problematic.
This misalignment leads to multimodal hallucination, which is, of course, problematic.
Unlike text-only hallucinations, which can sometimes be cross-referenced against external knowledge sources, visual hallucinations can only be verified by comparing model outputs directly against the source image — making them harder to detect at scale.
A model may describe attributes that aren’t present in an image or infer relationships that don’t exist, all while maintaining a high degree of confidence in its language.
The reality check for enterprises
These limitations are increasingly visible in enterprise applications.
In document processing, for example, models may extract text accurately but misinterpret layout or hierarchy, leading to incorrect conclusions. Areas such as medical imaging are even higher-stakes, where even small errors in reasoning can have significant consequences, particularly if a model misjudges the relationship between observed features.
Video analysis introduces even greater complexity because, unlike static images, video requires models to reason across time. Events unfold over sequences of frames, and critical information may be distributed across long intervals. Models must track changes, infer causality, and distinguish signal from redundancy, all while managing significantly larger data volumes.
As emerging video-native systems scale to handle temporal data, they encounter issues such as token explosion, memory constraints, and difficulty maintaining long-range dependencies. These challenges directly affect the model’s ability to reason coherently about dynamic environments.
Rethinking training and evaluation
Improving multimodal reasoning will take more than just building bigger models or tweaking their architecture. It calls for a different way of thinking about how these systems are trained and tested.
A major shift underway is the rise of multimodal instruction tuning. Instead of relying only on image-caption pairs, newer methods use structured tasks that force models to reason. This might mean answering questions about an image or explaining what’s happening in a scene, for example. That way, the model doesn’t just describe what it sees; it learns to interpret meaning and context.
But better training isn’t enough on its own. Evaluation must evolve as well. Traditional benchmarks like visual question answering still measure correctness at the surface level. They tell us whether a model got the ‘right’ answer, but not if it arrived there through coherent reasoning or if its explanation fits the visual input.
New evaluation methods are starting to fix this by adding more human perspective. Some benchmarks now assess how closely a model’s description of an interface or image matches human perceptions of clarity, context or relevance. The goal is to move beyond simple accuracy toward a deeper measure of reasoning quality.
The role of data and human oversight
If there’s one constant across every breakthrough in multimodal AI, it’s data. To build high-quality multimodal datasets, you need to ensure that images, annotations and tasks are all aligned.
This often means involving people directly. Humans still play a crucial role in defining what’s true, evaluating model behavior, and flagging cases where automation fails. Human oversight is an essential safeguard as multimodal systems become more complicated.
Subject-matter expertise is especially needed where context or ambiguity are nuanced, such as in medicine, design or social interaction. Images can evoke multiple technically valid interpretations, and human annotators with relevant domain expertise remain the most reliable means of distinguishing among them.
If a model doesn’t have the benefit of thoughtful annotation and feedback, it may latch onto shortcuts that won’t hold up when it faces real-world scenarios.
From scaling to grounding
For years, progress in multimodal AI has been measured by scale: ‘larger’ models, ‘more’ data, ‘bigger’ capabilities. But scale alone won’t guarantee that a system genuinely understands.
The next stage will hinge on grounding training models to anchor their outputs to what they actually perceive in the input, rather than defaulting to statistical patterns acquired during pretraining. Evaluations should reward real-world accuracy rather than performance on narrow benchmarks.
Multimodal AI has huge potential to change how organizations work with data, from automating complex tasks to uncovering insights that might be hidden within unstructured content. But fully realizing that promise means helping machines move from seeing to truly reasoning.
Until that happens, understanding images and video will remain one of the toughest nuts to crack on the path toward genuinely intelligent systems.
