How to pass message metadata with StreamingTextResponse to useChat?

206 views Asked by At

I'm trying to pass message metadata with StreamingTextResponse and to read it on the client side with useChat.

This is the sample code I'm using:

import {NextRequest, NextResponse} from "next/server";
import {experimental_StreamData, Message as VercelChatMessage, StreamingTextResponse} from "ai";
import { ChatOpenAI } from "langchain/chat_models/openai";
import {PromptTemplate} from "langchain/prompts";
import {BytesOutputParser} from "langchain/schema/output_parser";

export async function POST(req: NextRequest) {
  try {
    const body = await req.json();
    const messages: VercelChatMessage[] = body.messages ?? [];
    const currentMessageContent = messages[messages.length - 1].content;

    const model = new ChatOpenAI({
      temperature: 0,
      streaming: true,
    });
    const prompt = PromptTemplate.fromTemplate(`
      You are a cheerful bot that should respond to all questions in an upbeat manner.
      
      User input: {input}
      
      AI response:
    `);
    const outputParser = new BytesOutputParser();
    const chain = prompt
      .pipe(model)
      .pipe(outputParser);
    const stream = await chain.stream({input: currentMessageContent});

    return new StreamingTextResponse(stream);
  }
  catch (e: any) {
    return NextResponse.json({ error: e.message }, { status: 500 });
  }
}

I tried using experimental_StreamData() and passing the data object as the third argument to StreamingTextResponse():

const data = new experimental_StreamData();
data.append({text: new Date().toISOString()});
data.close();

return new StreamingTextResponse(stream, {}, data);

However, this just stringifies the data and prepends it to the response message.

What should I change here to pass the data object separately from the response?

Thanks!

0

There are 0 answers