protologue

Multimodal Prompting

Also called vision prompting, image prompting, visual prompting.

Multimodal prompting is prompting a model with more than one kind of input, such as images, documents, audio, or video together with text, so the model reasons over all of them.

Description

Text-prompting techniques largely carry over, but some practices are specific to non-text inputs. Provider guidance recommends placing images before the question, labeling multiple images so they can be referred to, and sending legible, uncompressed images. Models remain weaker at precise counting, small text, and spatial coordinates, so high-stakes visual answers need checking.

Example

Image 1: [receipt photo]. Image 2: [receipt photo]. List the total on each receipt and say which is larger.

Sources

  1. Yang et al. (2023). The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision).
  2. Anthropic (2024). Vision.

Cite this entry

Protologue. (2026). Multimodal Prompting. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0111). https://protologue.com/t/multimodal-prompting/

BibTeX
@misc{protologue_multimodal_prompting,
  title = {Multimodal Prompting},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0111},
  url = {https://protologue.com/t/multimodal-prompting/}
}

Markdown JSON