How to create an AI-based REST API endpoint using Node.js and Express that allows for real-time streaming of LLM-generated responses?

You want to build a chatbot app that can:

1. Receive input: Users send text messages via your React frontend (e.g., a chat window).
2. Send requests to your LLM backend: Use the Vercel AI SDK to integrate with a language model (like Google Gemini, Open AI) and stream the generated responses in real time.
3. Display responses on the client-side: Your React app should update with each new response.

streaming anything as response using nodejs is very easy, below is the fundamental code for streaming.

js
1// sample code for streaming response from nodejs express.
2function StreamToResponse(request, response) {
3  response.writeHead( 200, {
4    "Content-Type": "text/plain; charset=utf-8",
5  });
6    try {
7      for (let i = 0; i < 10; i++) { 
8         response.write(i); // Write to the response as we receive data
9        }
10    } catch (error) {
11      console.error("Error: ", error); 
12      throw error;
13    } finally {
14      response.end();  // End stream after all data is written
15    } 
16
17}

Using Vercel AI SDK you can implement in NodeJS with express framework like this :

js
1import { google } from "@ai-sdk/google";
2import { streamText, convertToCoreMessages } from "ai";
3import express from "express";
4
5const app = express
6
7app.post('/generate', async (req, res) => {  
8    try {   
9        const messages = req.body.messages; 
10        const result = await streamText({
11            model: google("your_model"),
12            messages: convertToCoreMessages(messages),
13        });
14  
15        result.pipeDataStreamToResponse(res); // this ai sdk package will take care of streaming response to client and handle according to that.
16    } catch (err) {
17        console.log(err); 
18        res.send(err);
19    }
20});
21
22module.exports = router;