Vision Language Models: How Multimodal AI Actually Works

What is a vision language model?

A vision language model (VLM) is a neural network that takes images and text in the same input and produces text as output, by converting the image into token embeddings that sit in the same representation space a language model already uses for words. Once an image has been turned into a sequence of vectors that look, mathematically, like word embeddings, the language model can attend over the image and the text together and reason about both in a single pass.

That one design decision is why VLMs improved so rapidly. Nobody had to invent a new reasoning engine. Get the pixels into the language model's vocabulary space accurately enough, and its existing capabilities — instruction following, reasoning, structured output — transfer to the visual domain largely for free.

This article is the mechanical explanation; for the business view, see the companion piece on the applications of vision language models.

How an image becomes tokens: patches, encoders and embeddings

Text tokenisation is discrete: a string maps to integers from a fixed vocabulary. Images have no such vocabulary, so VLMs build one on the fly — and almost every practical limitation can be traced back to one of these four steps.

1. Patching. The image is divided into fixed-size squares, commonly 14×14 or 16×16 pixels. A 336×336 image at 14-pixel patches gives you 576 patches, each of which is flattened into a vector.
2. Patch embedding. A learned projection maps each patch to the encoder's hidden dimension, plus a positional embedding so the model knows where it came from — the visual equivalent of a token embedding table.
3. Vision encoding. The sequence goes through a vision transformer. Self-attention lets the patches exchange information, so the outputs encode context — not "dark pixels here" but "the stem of a letter in a table cell".
4. Projection into language space. A small trained network maps the encoder output into the language model's embedding dimension. These become image tokens, interleaved with your text tokens.

The language model just receives embeddings — some from words, some from an image — and predicts the next token. The upshot is that images cost tokens: a single high-resolution page can consume as many tokens as several pages of text, so count your image tokens first, because they dominate both the bill and the latency.

Projection layers, adapters, and early versus late fusion

The connector between vision and language is where architectures part company, and it decides what the model is capable of.

The connector

• Linear or MLP projector. A one- or two-layer network that maps each visual vector to a language embedding. Very cheap to train and preserves spatial detail, though the token count scales directly with resolution.
• Query-based resampler. Learned query vectors cross-attend over the encoder output and compress hundreds of patches into a fixed token budget. Predictable cost, but it throws away fine detail, which hurts on dense text.
• Pooling or pixel shuffle. Merge neighbouring patch embeddings before projection, cutting the token count by four times or more. The pragmatic compromise for high-resolution input.

Where fusion happens

Late fusion keeps the towers separate. Contrastive models are the extreme case: image and text encoders trained so that matching pairs end up close together in a shared space. Nothing is generated; you just compare vectors. Superb for search, useless for reading a chart.

Early fusion feeds the visual tokens into the language model's input sequence, so every layer processes vision and text together. That's what makes genuine visual reasoning possible — following a line to its label, reading a value from a table and doing arithmetic with it. A middle option inserts gated cross-attention into a frozen language model, which preserves its text behaviour exactly and handles interleaved images well, at the cost of extra parameters and a harder training recipe.

The main VLM architecture patterns, and what each is good at

Most production systems use more than one: a contrastive encoder for cheap retrieval over millions of assets, a native multimodal model for the pages that actually need reading, and a small fine-tuned model for the extraction that runs ten thousand times a day.

Native multimodal pretraining versus bolted-on adapters

That distinction is invisible from the API signature. Both models take an image as input; only one of them genuinely learned to see as it learned to think.

The adapter route trains a connector between a finished language model and a finished vision encoder, on image-caption pairs and then visual instruction data. It's efficient, reproducible on quite modest hardware, and accounts for most of the open-weight ecosystem. Its limitation is structural: the language model's world was built from text alone, and the encoder was trained on photographs, so it's excellent at objects and mediocre at a spreadsheet.

Native multimodal pretraining interleaves images, text and often video and audio from the very beginning — on web pages, documents and screen recordings, as they naturally occur together. The model learns that a caption describes the figure above it and that a table header governs the column below. That's the capability gap you notice immediately on documents, charts and dense screenshots.

The buying guidance follows. For natural images — products, damage, places — a fine-tuned adapter model is often the better economic choice. For documents, forms, charts and screens, start natively multimodal, and only move down-market once a specialist has matched it on your data. The same logic applies to foundation model selection in general: the pretraining data sets the ceiling, and fine-tuning decides how close you get.

Resolution, tiling, and why dense documents broke early VLMs

The first generation of VLMs resized every input to a fixed square — 224×224 or 336×336 pixels. That was fine for "is there a dog in this photograph?" and disastrous for an A4 invoice, where 9-point text becomes a smear before the encoder even sees it. Teams concluded that VLMs couldn't read. In fact they could; the resize step had already destroyed the evidence.

How tiling fixed it

• Split, then encode. The page is divided into tiles at the encoder's native resolution, and each tile is encoded independently. Small text survives because it's never downscaled.
• Keep a global view. A downscaled image of the whole page is encoded alongside the tiles, so the model retains the page-level layout. Good schemes adapt the grid to the shape of the page.
• Budget the tokens. Tiling turns accuracy into a cost dial: more tiles, better reading, a bigger bill. Set the budget deliberately rather than discovering it in an invoice.

Why OCR-free understanding changed the game

The traditional document pipeline was a chain of specialist stages — cleanup, OCR, layout analysis, table reconstruction, key-value matching, then business logic — each with its own error rate, and those errors compounded. A capable VLM collapses that entire chain: you give it the page and the question and it answers, reading the text, interpreting the layout and understanding the table in one go. It also handles things OCR never could — a handwritten note in the margin, a logo that identifies the vendor, a chart with no data table behind it. Combine it with retrieval and you get multimodal RAG over document archives, where a citation can point to a region of a page.

The caveat: OCR-free is not verification-free. VLMs transcribe long strings of digits imperfectly, and a misread account number is a serious error. Keep a checksum or a targeted cross-check on fields where exactness is non-negotiable.

Grounding, bounding boxes and structured visual output

A description of an image is often not what you need. You need to know where. Grounding turns a VLM from a captioner into a component you can actually wire into software.

Modern VLMs output coordinates as text. Trained on data that pairs phrases with regions, they produce normalised box coordinates, so "find every damaged pallet and box each one" returns machine-readable regions rather than prose. The same mechanism yields point coordinates for UI elements, which is what makes screen-operating AI agents possible: the agent needs to know not just that a "Submit" button exists, but where to click it.

• Open-vocabulary detection. Ask for "corroded fittings" without having trained a detector for that class. Precision trails a purpose-built detector, but the setup cost is close to zero, and the class list can change on a Monday.
• Region-grounded answers. Every extracted field carries the coordinates it came from, so a reviewer can see exactly where it came from on the page. Constrain the response to a schema too, and a missing value becomes explicit rather than invented.
• Know the limits. For counting small objects, tight-tolerance localisation or real-time frame rates, a classical detector still comes out on top. The two compose well: detect cheaply, then ask the VLM to interpret the crops.

Beyond images: documents, screens, video and audio

"Multimodal" today means far more than photographs, and the extension is architecturally boring — it's the same trick applied to new data.

• Documents and charts. Pages, forms, invoices and slides. The dominant enterprise use, and the one where native pretraining pays for itself fastest.
• Screenshots and UI. Screens are a distinct visual domain, with their own grammar of buttons, menus and modals. Models trained on screen data are the ones that underpin testing and computer-using agents.
• Video. Frames are sampled and encoded as image tokens carrying temporal position, so the model can reason across time. The constraint is arithmetic: an hour of video is an enormous token sequence, so systems sample adaptively and summarise hierarchically.
• Audio and speech. Audio is windowed into spectrogram patches, encoded and projected into the same embedding space — the same recipe. That's what lets one model hear a caller, read the document they're referring to and answer, all within a single context: the foundation of a capable voice AI system.
• Interleaved everything. The new capability is in mixing them: a ticket with screenshots, a PDF and a screen recording, all reasoned over as a single input rather than five disconnected pipelines.

Modality is a preprocessing decision rather than an architectural one, which is why sensors, time series and 3D data are all following the same path.

Why VLM benchmark scores mislead on your data

Every VLM release arrives with a table of benchmark wins. Those numbers barely predict how it will perform on your actual workload, and treating them as procurement evidence is a common and expensive mistake.

• Benchmarks test short answers. Public suites score a word against a reference. Your task is "extract 30 fields into a schema and flag the ones you're unsure about". Not the same skill.
• The images are clean and public. Benchmark images are well-lit and often web-sourced, so they may well be in the training data. Your input is a phone photo of a crumpled delivery note.
• Aggregates hide the failure you care about. A model that's excellent overall may be systematically poor on your one document type or language. The average tells you nothing about the tail, and it's the tail that generates the incidents.
• Nobody measures refusal, cost or latency. In automated workflows, a model that says "this field is illegible" beats one that's marginally more accurate but guesses confidently — and a model that wins by a point at four times the token cost loses in production.

The fix is unglamorous. Collect a hundred or two of your own real inputs — including the ugly ones people tend to exclude — and have a domain expert write the correct output for each. Run every candidate against that set, scoring field accuracy, refusal calibration, token cost and p95 latency. It takes days, costs almost nothing, and repeatedly overturns the leaderboard ranking — the discipline that an honest account of computer vision advantages and challenges has always demanded.

Hosted frontier VLM or open-weight model you fine-tune?

Assume you'll run more than one, and choose per workload. Hosted models are versioned and deprecated on the vendor's schedule, so pin a version and re-run your evaluations when it moves.

A tiered approach works well. Route most of the traffic to a small fine-tuned model, measure its confidence, and escalate only the uncertain cases to a frontier model. You keep the unit economics of one and the accuracy of the other, and the escalated cases become your next training set. Where residency is the constraint, a hosted model running inside your own cloud region — for example on Vertex AI and Gemini — often resolves the objection without any loss of capability.

How SoftmaxAI builds multimodal systems

We build our vision-language systems the same way we build everything else: a written, client-approved specification before any code, an agent-accelerated build led by senior engineers, and every decision that matters owned by a person.

• The spec names the inputs and the acceptable errors. Which document types, languages and capture conditions; which fields must be exact; what happens when the model is unsure. Multimodal projects fail on the ugly inputs that weren't specified, not on the clean ones.
• Evaluation set before model selection. We build the labelled set from your real inputs first, then benchmark the candidates against it. Model choice becomes evidence rather than preference, and it's re-run whenever a hosted version changes.
• Human review where it belongs. Grounded outputs with region coordinates, confidence thresholds and a review queue for cases needing a person. Automation that cannot be checked does not ship.
• Built to run in your estate. Deployed into your own cloud account with monitoring, drift detection and a retraining path — the ML, computer vision and MLOps work that keeps a model useful in year two. As a Google Cloud Partner we build natively on GCP where it makes sense.

If you're still not sure whether a VLM is the right answer at all, the AI Opportunity Sprint is a short, fixed-scope engagement that tests the idea against your real inputs before you commit to a build.