How to format a few-shot prompt for GPT4 Chat Completion API?

Question

I'm trying to use the GPT4's chat completion API for the following prompt:

For each situation, describe the intent. Examples:


Situation 1: Devin gets the newspaper.

The intent of Situation 1: Devin intends to read the newspaper.

Situation 2: Jamie works all night.

The intent of Situation 2: Jamie intends to meet a deadline.

Situation 3: Sydney destroys Ryan.

The intent of Situation 3: Sydney intends to punish Ryan.

Situation 4: Lindsay clears her mind.

The intent of Situation 4: Lindsay intends to be ready for a new task.

Situation 5: Rowan wants to start a business.

The intent of Situation 5: Rowan intends to be self sufficient.

Situation 6: Lee ensures Ali’s safety.

The intent of Situation 6: Lee intends to be helpful.

Situation 7: Riley buys lottery tickets.

The intent of Situation 7: Riley intends to become rich.

Situation 8: Alex makes Chris wait.

The intent of Situation 8: Alex intends

As you can see, I want to complete the sentence that says "Alex intends". This prompt is intuitive for GPT3's Completion API where you only had to put one prompt that has all the few-shots examples.

However, I don't know what is the best practice to perform the same prompting with GPT4's ChatCompletion API. I've checked out https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb where they provided an example of how to do few-shot prompting, but my prompt is not "conversational" as you can see.

I'm not even sure whether the "name" parameter impacts the result's quality. Does anybody have an answer to this?

What I thought of so far is to format my prompt like this as the content from the above link instructed:

messages=[
        {"role": "system", "content": "For each situation, describe the intent. Examples:"},
        {"role": "system", "name":"Situation 1", "content": "Devin gets the newspaper."},
        {"role": "system", "name": "The intent of Situation 1", "content": "Devin intends to read the newspaper."},
        {"role": "system", "name":"Situation 2", "content": "Jamie works all night."},
        {"role": "system", "name": "The intent of Situation 2", "content": "Jamie intends to meet a deadline."},

...
        {"role": "system", "name":"Situation 8", "content": "Alex makes Chris wait."},
        {"role": "user", "name": "The intent of Situation 8", "content": ""},
    ]

Is this a proper way to do few-show with GPT4 ChatCompletion API? Please let me know if you have a better solution or explanations on why certain parts of my prompt needs work.

So far, I've simply put the original prompt into one user content, just like it is GPT3's Completion API:

messages=[
    {"role": "user", "content": "For each situation, describe the intent. Examples:


Situation 1: Devin gets the newspaper.

The intent of Situation 1: Devin intends to read the newspaper.

Situation 2: Jamie works all night.

The intent of Situation 2: Jamie intends to meet a deadline.

Situation 3: Sydney destroys Ryan.

The intent of Situation 3: Sydney intends to punish Ryan.

Situation 4: Lindsay clears her mind.

The intent of Situation 4: Lindsay intends to be ready for a new task.

Situation 5: Rowan wants to start a business.

The intent of Situation 5: Rowan intends to be self sufficient.

Situation 6: Lee ensures Ali’s safety.

The intent of Situation 6: Lee intends to be helpful.

Situation 7: Riley buys lottery tickets.

The intent of Situation 7: Riley intends to become rich.

Situation 8: Alex makes Chris wait.

The intent of Situation 8: Alex intends"}
]

It does work, but I was wondering if I can boost the API's performance if I follow a certain practice.

Answer 1

You might want to check out our GPT best practices guide which talks about how to do prompting effectively. The reality is that prompt engineering for tasks like this is much more art than science today. I suggest trying both approaches but my hunch is the one with each example as a user message / assistant response separately will perform better based on experiments I have seen.

Answer 2

I am not sure about the ChatCompletions API, but I know how few shots are generally done for the GPT-4 and GPT4-32K Apis.

You were actually pretty close to the few shot best practices in your try.

Template for best practices for few shots as far as I know:

messages=[

{"role":"system","content":"The_Prompt_You_Want_To_Give"},

#Few Shot starts here:

{"role":"user","content":"First_Example_Data_That_We_Send_To_GPT"},
{"role":"assistant","content":"First_Example_Answer_We_Expect"},

{"role":"user","content":"Second_Example_Data_That_We_Send_To_GPT"},
{"role":"assistant","content":"Second_Example_Answer_We_Expect"},

{"role":"user","content":"Third_Example_Data_That_We_Send_To_GPT"},
{"role":"assistant","content":"Third_Example_Answer_We_Expect"},

{"role":"user","content":"Fourth_Example_Data_That_We_Send_To_GPT"},
{"role":"assistant","content":"Fourth_Example_Answer_We_Expect"},
.....


#Finally send the data for which we want GPT to run the prompt and find solution

{"role":"user","content":"Content_We_Want_GPT_To_Process_Based_On_Prompt"}

]

So the above is the template for Few-Shot using openAI.

The part that u missed in your try is the role of assistant.

So the solution would be

messages=[
        {"role": "system", "content": "For each situation, describe the intent. Examples:"},
        {"role": "user", "content": "Devin gets the newspaper."},
        {"role": "assistant", "content": "Devin intends to read the newspaper."},
        {"role": "user",  "content": "Jamie works all night."},
        {"role": "assistant",  "content": "Jamie intends to meet a deadline."},

...
        {"role": "user", "content": "Alex makes Chris wait."},
         ]

I am not quite sure of the "name" parameter you have used, thats a new one for me so I have removed it. If you are sure about the name parameter keep it also in the body(messages).

Thanks