A Universal Roadmap for Prompt Engineering: The Contextual Scaffolds Framework (CSF)
A general purpose mental model for effective prompt engineering.
Introduction
In my recent articles, I explored a new prompt engineering approach for ChatGPT4 which I referred to as program simulation. The method showcased ChatGPT4’s impressive ability to emulate a program state consistently. These explorations heightened my appreciation for the nuanced power of language — the weight of words, semantics, and overarching context. This article explores some of these nuances and proposes a universal framework for prompt engineering, which I’ve christened “The Contextual Scaffolds Framework.” As we shall see, this framework generalizes effectively and seems capable of pulling techniques like Chain of Thought(CoT), Flattery/Role Assignment, Program Simulation and others under one umbrella. Additionally it provides an easy to use mental model for effective prompt crafting in a multitude of scenarios.
Into the World of Pragmatics
My efforts to examine the nuances of language more closely began with pragmatics, a branch of linguistics that examines how context shapes meaning interpretation. Two concepts in particular stood out:
- Implicature: When a speaker implies something without explicitly stating it, expecting the listener to infer the intended meaning.
- Presupposition: Assumptions or information that dialogue participants believe to be shared between the speaker and listener.
The central insight from these concepts is that words’ meanings extend well beyond their literal definitions. Consider the term “dog.” Beyond its basic definition, it carries a wealth of implicit information. We don’t explicitly state that dogs exist through time, move through space, eat, hear, bark, etc. The expectation is that listeners share this knowledge and extract the appropriate meaning(s) given the context at hand. Every linguistic expression, be it a word or novel, emits a “meaning aura” — a blend of implicit definitions, implicatures, sentiment, and connotations. These “meaning auras” can additionally vary in density, complexity and clarity and are often situation dependent.
Large Language Models (LLM) and “Meaning Auras”
LLMs in some senses systemize the production of “meaning auras” in a conversational dialogue with humans. But at their core these models are merely making word by word predictions. Is it possible that they have implicitly modeled the interplay of “meaning auras”? How might we measure that? While answering these questions would necessitate in-depth research, our brief foray into pragmatics does offers some immediate practical applicability and can serve as an important building block for a universal prompt engineering framework.
The Contextual Scaffolds Framework (CSF)
As many have pointed out previously, effective prompt crafting for models like ChatGPT-4 hinges on context. But we also need to take into account what our expectations are for the model output and how the model should “operate” in order to meet those expectations. While bearing in mind the concept of “meaning auras” let’s examine an approach where the context of a prompt is broken down categorically. Let’s refer to these categories as “scaffolds” and specify two that are broadly applicable.
Expectational Context Scaffold — Encompasses the user’s aspirations, intent, objectives, and specifics of the situation at hand. If the user’s personal context is relevant it should be factored in as well.
Operational Context Scaffold — Establishes the AI’s operational parameters. It defines the model’s role, techniques to employ, required external data, and the extent of its autonomy and discretion.
Here is a straightforward visual representation of the framework:
Now, let’s see this approach in action with a prompt for ChatGPT-4. We will focus on selecting language for our scaffolds that have semantically rich “meaning auras” and are likely to produce the output we are looking for.
Context Scaffolds Prompt
“My Expectational Context — Your goal is to help me write a story about artificially intelligent teddy bears. The audience for my story is adults. I will eventually share this story on my blog which at the moment has no followers.
Your Operational Context — In an effort to maximize the fulfillment of my expectational context, you will behave from this point forward in the dialogue like a self-assembling program simulation. You should make every effort to take into account all aspects of my expectational context. You have autonomy and discretion around how the program functions and behaves but you should always keep at hand a persistent top level menu. Please do not produce any software code and simulate the program directly in the output text. Once this prompt is received please proceed with the simulation.”
You will all get something a little different, but it should in most cases look something like this:
As you can see from the output, ChatGPT-4 launched a program simulation that has for the most part fulfilled my “Expectational Context.” I significantly influenced this by specifying language in both scaffolds that emit contextually rich “meaning auras.”
So far, we have defined a straightforward universal framework for prompt crafting. Techniques such as Flattery/Role Assignment, Few-shot, and CoT predominantly fall under the “Operational Context Scaffold." While I couldn’t identify techniques rooted solely in the “Expectational Context Scaffold,” most goal-driven prompt language typically fits into this scaffold. So, in some way, we all implicitly use the technique by default.
But is that it? Do we just wrap up the article here or are there some derivative insights? Let’s see if we can unpack this further…
Optimization of Context Scaffolds for Prompt Engineering
When engaging with a chosen LLM, our ultimate hope is that the model will produce output that meets or exceed our expectations. If we look at our scaffolds from an optimization perspective, we can envision a framework where the goal is to pinpoint one or more “Operational Contexts” that best fulfill the “Expectational Context.” For those wary of mathematical jargon, bear with me; this is a brief detour.
We could represent such a function as follows:
O = LLM(EC)
Where:
O is a set of optimal Operational Contexts (OCₙ),
LLM is the function embodied by the Large Language Model.
Each element in set O, say OCᵢ, represents a different optimal Operational Context for the given Expectational Context:
O={OC₁,OC₂,…,OCₙ}
Since our Operational Context and Expectational Context are multi-dimensional we should likely model them in a more detailed manner as vectors of attributes:
EC={e₁,e₂,…,eₙ}
OC={o₁,o₂,…,oₙ}
Finally the objective function could be expressed as maximizing the utility function U over all possible OCs in set O for a given EC:
These mathematical abstractions attempt to systematize the transformation of the Expectational Context into an Operational Context, while acknowledging the high likelihood of multiple optimal Operational Contexts for a given Expectational Context. We will look at the possibility of finetuning models using this type of framework in the future, but for now, let’s look at the practical implications of these ideas.
Suppose you have a good handle on how to articulate your Expectational Context Scaffold but are uncertain on what elements to include in your Operational Context Scaffold. Can ChatGPT-4 assist?
Let’s craft a prompt accordingly and see what we get back.
Open-Ended Operational Context Prompt
“My Expectational Context — Your goal is to help me write a story about artificially intelligent teddy bears. The audience for my story is adults. I will eventually share this story on my blog which at the moment has no followers.
Your Operative Context — In an effort to maximize the fulfillment of my expectational context, please suggest at least one but no more than five operational contexts that you could employ. You might suggest behaving as a person, a team, a type of organization, program, or system with one or more specific competences. You might suggest the inclusion of external data or request that you be provided training examples. You might also suggest the use of specific techniques or approaches. You can suggest any combination of the above elements . The operational contexts should be rank ordered from most optimal to least optimal. For each please provide a rationale that leads to their specific rank.”
True to form, ChatGPT4 provides us with 5 operational contexts; 2 are organizational entities, 1 is a system and 2 are individuals. The operational contexts have been ranked and a rationale for each ranking has been included. Consider the “meaning auras” of concepts like “Literary Think Tank” or “Historical and Sci-fi Research Institute”. I think you would agree they are context dense and rich with meaning and implicatures. With this approach we can arm ourselves with a plethora of operational contexts that we may otherwise have not happened upon on our own. And it lowers the barrier to effective prompt crafting by narrowing down our starting point to focus on articulating the “Expectational Context Scaffold.»
Conclusion
As we delve into the intricacies of pragmatics and the concept of “meaning auras”, it is evident that context, in its multifaceted form, holds the key to to optimizing our interactions with LLMs. The Contextual Scaffolds Framework (CSF) gives us a straightforward mental model that can help us articulate context very effectively. By differentiating between “Expectational” and “Operational” contexts, the CSF provides a clear path for aligning user expectations with the capabilities of models like ChatGPT-4. Additionally the CSF is extensible and as other scaffolds become relevant or necessary they can be added or subtracted as needed. Similarly each scaffold can be subdivided into component scaffolds that represent distinct features for a given context.
Thanks for reading and I hope you find the CSF a useful model for crafting prompts! I am in the midst of additional explorations so be sure to follow me and get notified when new articles are published. If you would like to discuss CSF or any of my other articles further, do not hesitate to connect with me on LinkedIn.
Unless otherwise noted, all images in this article are by the author.
