Considering recent studies where LLM interactions are structured involving complex prompting strategies and some kind of solution is built around LLM interactions, two phenomenons are taking place:

1. There are more LLM calls involved in the Generative AI approach or strategy. I have always maintained that the approach of offloading most if not all of the functionality to LLMs implies more calls. This gives rise to additional overhead and problems like inference time, cost and dependancy on a specific LLM.
2. In many of the instances a number of calls are made to the LLM and one or more LLMs is used to select the best response to use, or present to the user.

These approaches are aimed at leveraging LLMs to the maximum and avoiding efforts in the area of fine-tuning models, introducing any human-in-the-loop process or external data.

Understanding these scaling dynamics is also helpful to understand the limits of compound inference strategies.

However, more LLM calls enhance performance on easy queries but diminish it on hard ones.

Reaching Consensus

This paper investigates the impact of the number of Large Language Model (LLM) calls on the performance of compound systems used in language tasks.

These compound systems aggregate responses from multiple LLM calls.

The research examines this theoretically and empirically to understand the scaling laws governing the performance of such systems.

The research shows that, contrary to expectations, the performance of Voting Inference Systems initially increases and then decreases with the number of LLM calls across various language tasks.

Considering the image below, this speaks to what I described in the introduction. Where multiple instances of inference is run, and a pre-defined process is used to select the best response to use.

More LLM calls enhance performance on easy queries but diminish it on hard ones.

By understanding this, the study devises a method to compute the optimal number of LLM calls to maximise system performance, based on a small number of samples.

Experiments validate that this scaling law accurately predicts the performance of Voting Inference Systems and identifies the optimal number of LLM calls.

Considering the image above, as the number of LLM calls increases, the one-layer Voting Inference System exhibits a notable performance trend: it improves significantly on easy items while simultaneously deteriorating on hard items.

Specifically, while the system benefits from increased LLM calls in addressing easier questions more effectively, it struggles to maintain performance on the more challenging items, resulting in an overall nuanced performance pattern.