Hierarchical Adaptive Recursive Language Models

Abstract

Recursive Language Models (RLMs; Zhang, Kraska, and Khattab, 2025) treat the user prompt as an external variable inside a Python REPL and let the model emit code to answer the query. The emitted code can include recursive llm_query calls on substrings of the prompt. This paper examines the published RLM harness and identifies three structural inefficiencies: synchronous execution of independent sub-queries, a single level of recursion, and uniform application of the REPL even to inputs that fit in the base model's effective window.

I propose Hierarchical Adaptive Recursive Language Models (HARLM) as a design that addresses each issue while keeping the RLM interface unchanged. HARLM adds a DAG-based parallel scheduler, a learned router, a three-tier memory, and an information-weighted token budget.

Each component is analyzed formally: the scheduler against a parallel execution bound, the router against the scaling classes of Zhang and colleagues, the memory against an explicit context-size bound, and the allocator against a read-at-least-once lower bound on input tokens. The purpose of releasing the design before experiments is to make the analysis reviewable on its own terms.

Three observations about the reference RLM

Three structural properties stand out as inefficiencies whose cost, latency, or accuracy impact can be reasoned about without rerunning the original experiments.

1. Synchronous sub-query execution. Generated Python runs inside one REPL whose calls execute sequentially. Wall-clock time grows linearly in the number of independent calls even when they could run concurrently.
2. Single-level recursion. Sub-calls invoke a base model, not another RLM. Aggregation is pushed onto one fan-out layer, which can put the aggregator back into the long-context regime.
3. Uniform inference strategy. The same REPL orchestration is used whether or not the input fits in the base model's effective window, adding overhead to short items.

HARLM architecture

HARLM extends RLMs with four integrated components. The scheduler relaxes the synchronous critical path. Hierarchical memory relaxes depth-one aggregation pressure. The adaptive router removes the uniform-strategy overhead. The token allocator connects resource decisions across all three.

Parallel and speculative scheduling

When the HARLM agent emits multiple llm_query calls, a conservative AST walk builds a dependency DAG. Any call whose argument touches the output of another is declared dependent. Overestimating dependencies gives up parallelism but cannot introduce incorrect concurrent execution. Independent batches are dispatched with bounded parallelism.

For items where the best decomposition is ambiguous, the scheduler can run up to three candidate strategies in parallel and cancel pending ones when a strategy returns a high-confidence answer. This is a latency hedge, not a cost reducer.

Learned adaptive routing

The router maps prompt and query pairs to Direct, REPL-only, or Full HARLM mode. The proposed architecture is a T5-large encoder with a three-way classification head, mean-pooled token embeddings, and scalar features for input length, effective-window estimate, and task-class estimate.

The loss combines cross-entropy on oracle labels with cost and performance penalties. Misrouting up is penalized less than misrouting down. A router that occasionally over-routes costs more but preserves correctness.

Hierarchical memory

If recursion proceeds beyond one level, a root aggregation step sees an exponentially growing number of sub-results. HARLM's three-tier memory bounds the context presented to any single aggregation call independent of depth.

• Hot: an LRU cache of the most recent sub-results at full fidelity.
• Warm: older sub-results compressed by a smaller model while preserving entities, numerals, and relevant predicates.
• Cold: a vector index over remaining sub-results retrieved with hybrid dense and BM25 search.

Information-weighted token budgeting

Given a global token budget, the allocator distributes tokens to nodes in proportion to a score combining chunk size, learned information density, and query relevance. Density is a regression head on the router encoder. Relevance is a dot product between cached sub-result embeddings and the query embedding. The allocator is a heuristic, not a claim of optimality.

Analytical study

Each component is analyzed against the observation it is designed to relax. The assumptions behind each argument are explicit, along with where they fail.

Scheduler versus the sequential bound

Partitioning K independent queries into batches of size P produces an approximate factor-P speedup over the sequential bound. This relies on synchronization overhead being dominated by network latency and the provider concurrency cap being at least P.

Router versus short-input overhead

The paper derives a condition on router accuracy under which routing reduces expected cost compared with always using full mode. When the cost ratio between full and cheaper modes is large, even a moderately accurate router should recover meaningful overhead.

Memory versus aggregation pressure

Context for any aggregation call is bounded by the hot capacity, compressed warm capacity, and top retrieved cold entries, independent of recursion depth and the total number of sub-results. The cost is information lost through compression and retrieval recall.

Token budget versus a read-at-least-once lower bound

Theorem 1, informal: under additive separability, no auxiliary oracle, and a per-call maximum context length W, any algorithm answering a task in scaling class C must perform enough base-model calls to consume every required input token at least once.

Proposition 6, informal: under additive separability, effective window W, and accurate routing, HARLM's expected input-token cost is within a log factor of the lower bound for constant, linear, and quadratic task classes. Long-range coreference is the assumption most likely to break this result.

Where the savings should come from

• The router should dominate cost reduction because it attacks the largest constant cost in the wrapper.
• The parallel scheduler should dominate latency reduction.
• The hierarchical memory should dominate accuracy improvements on quadratic-class tasks.

If ablations distribute damage uniformly, the component decomposition is wrong.

Evaluation plan

• Reproduce published RLM numbers across the original benchmark suite using two base-model families.
• Run full HARLM and ablations without the router, scheduler, memory, and speculative execution.
• Test transfer to newer model families while separating cost and latency ratios.
• Use at least three seeds per cell, bootstrap confidence intervals, publish raw scores, and state concurrency caps and pricing.
• Hand-label failure modes into sub-model, aggregation, decomposition, and routing errors.
• Release the router checkpoint, scheduler, memory implementation, and evaluation harness with the empirical paper.

Limitations

• Additive separability fails for tasks with heavy long-range coreference.
• Training the router requires oracle labels produced by running all three modes.
• Warm-tier compression is lossy and risky when verbatim strings matter.
• Provider concurrency caps bound scheduler speedup.
• Hardware-aware scheduling is not included.
• The analytical predictions have not yet been checked against empirical measurements.

Conclusion

HARLM is a narrow extension of the RLM idea. Synchronous sub-queries map to the parallel scheduler, single-level recursion maps to hierarchical memory, and uniform inference maps to the router. The analysis produces a falsifiable prediction about which component should dominate each metric. Whether that prediction holds remains an empirical question.

Citation

Mure, L. (2026). HARLM: Hierarchical Adaptive Recursive Language Models, A Design Study. SSRN. https://dx.doi.org/10.2139/ssrn.6397658