How to Use LangChain for Building AI-Powered API Endpoints
🔍 Want the best deal? Check current prices and availability.
Compare Prices →When you buy through links on our site, we may earn a commission.
How to Use LangChain for Building AI-Powered API Endpoints
So you want to expose an LLM through a REST API. Maybe you're building a support bot, a content summarizer, or something that needs to talk to a language model without living inside a notebook. You've probably heard of LangChain — it's the most popular framework for wiring up LLM applications — but there's a lot of confusion about whether it's actually worth the learning curve.
Short answer: yes, for anything beyond a single openai.ChatCompletion.create() call.
In this langchain tutorial, I'll walk you through building real AI API endpoints with LangChain and FastAPI. We'll cover setup, basic chat endpoints, streaming, memory, and RAG. I'll also give you an honest comparison with the alternatives so you know exactly what you're getting into. This is an llm application guide, not a marketing pitch — we're going to get our hands dirty.
What Actually Is LangChain?
LangChain is an open-source framework (MIT licensed) that gives you building blocks for LLM applications. Instead of hand-rolling every prompt template, every retry, every bit of memory management, you get composable pieces that snap together.
The core abstraction is called a chain — a sequence of steps that processes input and produces output. In modern LangChain (0.3+), you write chains using LCEL (LangChain Expression Language), which uses the | operator to pipe components together. It looks like this:
chain = prompt | llm | output_parser
That's it. Each component takes the output of the previous one and feeds it to the next. It's deceptively simple, and it's what makes LangChain worth using for anything non-trivial.
LangChain also ships with integrations for hundreds of models, vector stores, and tools. You can swap out OpenAI for Anthropic, Google, or a local Ollama model by changing a single line. That flexibility is the main reason developers reach for it.
Prerequisites
Before we start, Make sure you have:
- Python 3.9 or newer
- An OpenAI API key (or another provider's key — the pattern is the same)
- Basic familiarity with FastAPI and async Python
You can find the official LangChain docs for reference, but I'll cover everything you need here.
Setting Up Your Environment
Create a fresh directory and set up a virtual environment:
mkdir langchain-api
cd langchain-api
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
Now install the packages we need:
pip install langchain langchain-openai langchain-community fastapi uvicorn python-dotenv
For RAG later, you'll also want:
pip install faiss-cpu pypdf
Create a .env file with your API key:
OPENAI_API_KEY=sk-your-key-here
Building Your First Endpoint
Let's start with a simple chat endpoint. We'll create a FastAPI app that accepts a message and returns a response from GPT-4o-mini.
Create a file called main.py:
from fastapi import FastAPI
from pydantic import BaseModel
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
app = FastAPI(title="LangChain AI API")
class ChatRequest(BaseModel):
message: str
session_id: str = "default"
class ChatResponse(BaseModel):
response: str
session_id: str
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.7)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful assistant."),
("human", "{message}")
])
chain = prompt | llm | StrOutputParser()
@app.post("/chat", response_model=ChatResponse)
async def chat(request: ChatRequest):
response = await chain.ainvoke({"message": request.message})
return ChatResponse(response=response, session_id=request.session_id)
Run it with:
uvicorn main:app --reload
Then hit it with curl:
curl -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"message": "What is the capital of France?"}'
You'll get back a JSON response with the answer. That's your first AI-powered API endpoint.
Notice how the chain is defined: prompt | llm | StrOutputParser(). The prompt formats the input, the LLM generates a response, and the output parser converts the message object into a plain string. Clean and composable.
Adding Streaming for Real-Time Responses
Nobody likes waiting three seconds for a spinner. If you're building anything user-facing, you want tokens to stream back as they're generated. FastAPI supports this natively with StreamingResponse.
Here's how to add streaming to your endpoint:
from fastapi.responses import StreamingResponse
from fastapi import Request
import json
@app.post("/chat/stream")
async def chat_stream(request: Request):
body = await request.json()
message = body.get("message", "")
async def generate():
async for chunk in chain.astream({"message": message}):
yield f"data: {json.dumps({'token': chunk})}\n\n"
return StreamingResponse(
generate(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "Connection": "keep-alive"}
)
Now the client receives Server-Sent Events (SSE) as tokens arrive. This is the same pattern used by ChatGPT and Claude. You can consume it from the frontend with EventSource or fetch with a ReadableStream.
Adding Memory and Conversation Context
Here's where LangChain starts to earn its keep. Stateless chat is fine for a demo, but real applications need context. Users expect the assistant to remember what they said five messages ago.
LangChain provides RunnableWithMessageHistory for this. It stores message history by session ID and injects it into the prompt automatically.
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.messages import HumanMessage, SystemMessage
store = {}
def get_session_history(session_id: str):
if session_id not in store:
store[session_id] = InMemoryChatMessageHistory()
return store[session_id]
Keep the prompt simple — history gets injected automatically
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful assistant."),
("placeholder", "{history}"),
("human", "{message}")
])
chain = prompt | llm | StrOutputParser()
chain_with_history = RunnableWithMessageHistory(
chain,
get_session_history,
input_messages_key="message",
history_messages_key="history"
)
@app.post("/chat/memory", response_model=ChatResponse)
async def chat_with_memory(request: ChatRequest):
response = await chain_with_history.ainvoke(
{"message": request.message},
config={"configurable": {"session_id": request.session_id}}
)
return ChatResponse(response=response, session_id=request.session_id)
Now if you send "My name is Alex" followed by "What's my name?", the second call will correctly answer "Alex". The history is stored in memory on the server, keyed by session_id.
A word of caution: InMemoryChatMessageHistory stores everything in RAM. For a production deployment, you'll want to back this with Redis or Postgres. LangChain has integrations for both, and swapping them in is straightforward.
Building a RAG Endpoint
Retrieval-Augmented Generation (RAG) is the most common real-world use case for AI APIs. You give the model access to your own documents so it can answer questions about your specific data without retraining.
Here's a minimal RAG setup with LangChain,
🔍 Want the best deal? Check current prices and availability.
Compare Prices →