The fastest way to get a response from a large language model isn’t by asking it to be faster, but by making it ask itself a question it already knows the answer to.
Let’s say you have a simple prompt:
{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
]
}
When this hits an API like OpenAI’s, it goes through a whole pipeline. The model needs to understand your instruction, fetch its knowledge about France, and then formulate the answer. This takes time.
Now, consider this:
{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France? The answer is Paris."}
]
}
This looks like cheating, and in a way, it is. You’re providing the answer within the prompt itself. When the model receives this, it doesn’t need to compute the answer. It just needs to confirm that the provided answer is indeed correct, or if it’s supposed to parrot it back. For many use cases, this confirmation is much faster than generating the answer from scratch. The model’s internal process shifts from "reasoning" to "verification."
How It Works: The Illusion of Speed
Large language models, at their core, are sophisticated pattern-matching machines. When you ask "What is the capital of France?", the model traverses its internal representations, looking for the strongest association between "capital" and "France." This involves activating various neural pathways and performing complex calculations.
When you provide the answer, you’re essentially short-circuiting this process. The model’s task becomes: "Does the input 'Paris' align with the query 'capital of France'?" This is a much simpler lookup or a direct conditional check. It bypasses the computationally intensive generation phase.
Imagine you ask a human: "What is the capital of France?" They might pause for a second to recall. Now, imagine you ask: "What is the capital of France? Is it Paris?" They can likely confirm "Yes, it’s Paris" almost instantaneously. The LLM behaves similarly.
Practical Applications: When This is a Game-Changer
This technique is particularly useful in scenarios where:
- Real-time interaction is critical: Chatbots in live customer service, interactive games, or any application where a millisecond matters.
- The answer is predictable or known: For FAQs, simple data lookups, or standardized responses.
- You need to guide the model’s output precisely: If you want the model to always respond with a specific phrase or format, embedding it is the most reliable way.
- Reducing API costs: Faster processing often translates to lower token usage and reduced inference costs.
Let’s see this in action with a hypothetical API call.
Scenario 1: Standard Prompt (Higher Latency)
curl https://api.example-llm.com/v1/completions \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{
"model": "gpt-3.5-turbo",
"prompt": "What is the capital of France?",
"max_tokens": 10,
"temperature": 0.7
}'
Expected Response Time: Varies, but could be 1-3 seconds depending on load and model. Model’s Task: Generate "Paris" based on its training data.
Scenario 2: Prompt with Answer (Lower Latency)
curl https://api.example-llm.com/v1/completions \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{
"model": "gpt-3.5-turbo",
"prompt": "What is the capital of France? The answer is Paris.",
"max_tokens": 10,
"temperature": 0.7
}'
Expected Response Time: Varies, but could be significantly less, perhaps 0.5-1.5 seconds. Model’s Task: Confirm or parrot "Paris" as the answer to the question.
The max_tokens and temperature parameters still play a role, but the initial computation to get any token out (time-to-first-token) is dramatically reduced when the model isn’t doing heavy lifting.
Controlling the Output
While you’re providing the answer, you can still influence how the model presents it.
- Confirmation: If you add "Is it Paris?", the model might respond "Yes, it is Paris."
- Direct Output: If you add "The answer is Paris.", the model might simply output "Paris."
- Slight Variation: Adding "The capital of France is Paris." might lead to the model confirming or slightly rephrasing.
The key is to make the provided answer part of the context or the expected output format, rather than a piece of information the model needs to discover.
The Counterintuitive Levers
The most effective way to use this "pre-answered" prompt isn’t just to dump the answer at the end. It’s to integrate it into the prompt in a way that frames the LLM’s task as verification or completion. For instance, instead of "What is X? The answer is Y", you could use "Given that X is Y, confirm this fact and provide one related detail." This subtly shifts the model’s internal computation from pure retrieval to a more structured, but still faster, verification and elaboration task. It leverages the model’s ability to understand nuanced instructions while still benefiting from the pre-supplied knowledge.
This technique directly targets the initial computational load, making it a powerful tool for optimizing latency in LLM applications.
The next step in optimizing LLM interaction involves understanding how to batch requests efficiently when dealing with multiple such pre-answered prompts.