protologue

RULER

RULER is a synthetic long-context benchmark that extends needle-in-a-haystack retrieval with harder tasks, such as multiple needles, multi-hop tracing, and aggregation, at configurable context lengths, to measure how much of its context window a model can actually use.

Description

Its creators found that many models performed well on simple needle retrieval but degraded well before their advertised context length on the harder tasks. It is a standard reference for a model's effective, as opposed to claimed, context size.

Sources

  1. Hsieh et al. (2024). RULER: What's the Real Context Size of Your Long-Context Language Models?.

Cite this entry

Protologue. (2026). RULER. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0211). https://protologue.com/t/ruler/

BibTeX
@misc{protologue_ruler,
  title = {RULER},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0211},
  url = {https://protologue.com/t/ruler/}
}

Markdown JSON