An engineering take on the Agent memory and state.
In production-grade Agentic AI systems, state management and memory are the hardest problems to solve. Unlike traditional stateless microservices where each request is independent and idempotent, agentic workflows are inherently stateful, long-running, and non-deterministic. A single user conversation can span dozens of turns, invoke multiple specialized agents, trigger tool calls, and require human intervention at critical decision points. Losing context mid-flow or allowing concurrent mutations destroys trust and correctness.
In our production environment, we learned this the hard way. Early prototypes treated agents like stateless API handlers. Under realistic load, conversations fractured, tool results were lost, and partial agent scratchpads disappeared on worker restarts. The shift to explicit, durable, and thread-safe state management became non-negotiable for enterprise SLAs.
The State Problem in Multi-Agent Systems
Multi-agent architectures (supervisor + specialist agents) compound the problem. Each specialist maintains its own scratchpad, tool history, and intermediate reasoning. The supervisor must orchestrate across them while preserving a coherent global conversation context. Without careful design, you end up with either massive prompt bloat (sending entire history every turn) or silent context loss.
We adopted Redis-backed conversation memory as the backbone. Redis serves three roles:
- Persistent store for LangGraph checkpoints (full graph state including messages, tool calls, and interrupt metadata).
- Short-term memory for active conversation turns (via RedisChatMessageHistory or custom key patterns like
conv:{thread_id}:messages). - Tool output cache with fine-grained TTL to avoid repeated expensive calls (e.g., database lookups or external API results).
Why Redis over Postgres? Postgres offers strong durability and complex querying, but introduces unacceptable latency for the hot path of every agent turn. In our benchmarks, Redis delivered sub-millisecond p99 get/set for state blobs up to 256 KB, while Postgres hovered at 8–15 ms even with connection pooling and prepared statements. Redis also provides native TTL, atomic operations, and pub/sub—features we use for cache invalidation and cross-worker signaling. For long-term archival or audit logs we still replicate to Postgres or S3, but the operational state machine lives in Redis.
TTL strategy is critical for cost and correctness. We apply a default 48-hour TTL on active conversation keys, with sliding expiration on every user message. High-value workflows (compliance reviews, financial approvals) receive longer or indefinite TTLs with explicit archival hooks. This balances context retention against Redis memory pressure and infrastructure cost. Under peak load we observed ~35% memory reduction versus no TTL, without measurable impact on user experience.
Concurrency & Thread Safety in Python/FastAPI
FastAPI endpoints handling thousands of concurrent requests cannot afford to instantiate heavy agent objects (LLM clients, tool bindings, prompt templates, LangGraph graphs) on every request. The first version of our platform did exactly that. Load tests with 500 concurrent conversations revealed p99 latency spikes above 7 seconds purely from object construction and tool schema binding before the first LLM call even occurred.
The architectural solution is a thread-safe singleton for the core Agent class using the double-checked locking pattern. Because Uvicorn workers can use multiple threads (via ThreadPoolExecutor for sync LLM SDKs) and because initialization is expensive and non-idempotent, we must guarantee exactly-once initialization even under race conditions.
import threading
from typing import Optional, Any
class ThreadSafeAgent:
_instance: Optional["ThreadSafeAgent"] = None
_lock: threading.Lock = threading.Lock()
_initialized: bool = False
def __new__(cls, *args: Any, **kwargs: Any) -> "ThreadSafeAgent":
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
return cls._instance
def __init__(self, llm_config: dict, tools: list, redis_url: str) -> None:
if self._initialized:
return
with self._lock:
if self._initialized:
return
# One-time heavy initialization
self.llm = self._build_llm(llm_config)
self.tools = self._bind_tools(tools)
self.graph = self._build_langgraph(self.llm, self.tools)
self.redis = self._init_redis(redis_url) # connection pool
self._initialized = True
# ... build_langgraph, checkpoint saver wiring, etc.We mount this singleton via FastAPI lifespan events and expose it through dependency injection. Combined with Redis-backed checkpoints, any worker can resume any conversation without local state.
Architecting Human-in-the-Loop (HITL) Workflows
Fully autonomous agents remain a non-starter for enterprise workloads. Compliance requirements (SOX, GDPR, financial controls), risk of hallucinated actions, and the need for audit trails force human approval at sensitive nodes—refunds, contract changes, external API commits, or PII handling.
LangGraph’s interrupt mechanism provides the cleanest way to implement this without turning the entire system into a polling nightmare. When a node calls interrupt(...), LangGraph persists the current graph state (including the full message history and scratchpad) to the configured checkpointer (Redis in our case) and returns control to the caller with an __interrupt__ marker. The HTTP request completes; no thread is blocked indefinitely.
The UI then presents the pending decision. On human approval (or rejection + feedback), a dedicated resume endpoint reconstructs the config with the same thread_id and invokes the graph again with the human payload. LangGraph replays from the last checkpoint, injects the human response, and continues execution. Because state lives in Redis, any worker can resume the conversation—even if the original worker died.
from langgraph.types import interrupt
from langgraph.graph import StateGraph
def approval_node(state: AgentState):
decision = interrupt({
"question": "Approve refund of ${amount} to {customer}?",
"context": state["last_tool_result"]
})
return {"human_decision": decision["approved"], "feedback": decision.get("feedback")}We surface pending interrupts via a lightweight approval queue API and use OpenTelemetry spans to trace the full “pause → human think time → resume” lifecycle for auditability.
Guardrails as Middleware
Security and auditability cannot be bolted on after the LLM call. We intercept every prompt and tool input through a multi-layer LangChain middleware pipeline before it reaches the model.
The pipeline executes as a chain of callables (or custom Runnable wrappers) in a defined order:
- PII redaction / tokenization
- Profanity and toxicity filter (
oss libraries+ custom allow/deny lists) - Topic and policy classifier (lightweight local model or rules)
- Output guard (post-LLM but pre-persistence)
Example integration with better-profanity:
from better_profanity import profanity
from langchain_core.runnables import RunnableLambda
def profanity_guardrail(prompt: str) -> str:
if profanity.contains_profanity(prompt):
# Log, sanitize, or raise PolicyViolation
raise ValueError("Content policy violation detected")
return prompt
guardrail_chain = RunnableLambda(profanity_guardrail) | RunnableLambda(pii_redact) | ...Because the middleware runs in-process before any network call to the LLM provider, we maintain low latency while satisfying enterprise audit requirements. Every guardrail decision is emitted as a structured log event with correlation IDs.
Real-Time UX via SSE Streaming
Users expect to see the agent “think” in real time. We expose Server-Sent Events (SSE) endpoints that stream three event types: thought (intermediate reasoning), tool_call / tool_result, and final_answer.
FastAPI’s StreamingResponse with media_type="text/event-stream" combined with LangGraph’s astream_events or astream yields progressive updates without buffering the entire response. The frontend (React + EventSource) renders thoughts, tool progress, and streaming tokens as they arrive. This keeps perceived latency low even when a supervisor → specialist → tool → HITL cycle takes 30–90 seconds.
Architectural Trade-offs
Two fundamental trade-offs shaped our final design.
In-memory state vs. Redis-backed state. Pure in-memory (process-local dict or lru_cache) offers zero network latency and trivial implementation. We rejected it because it fails horizontal scaling, loses state on any worker restart or rollout, and cannot support HITL resume from another pod. Redis adds 1–4 ms p99 round-trip but buys durability (AOF + RDB), shared visibility across workers, native TTL, and atomic operations. For our workload the latency tax was acceptable; we mitigated it further with client-side connection pooling and pipelining.
LangGraph interrupts vs. simpler fire-and-forget async patterns. Fire-and-forget (Celery/RQ + webhook callbacks) is simpler to implement and scales easily for purely autonomous flows. It breaks down for HITL because you lose the ability to cleanly pause and resume the exact point in the state machine while preserving full context. LangGraph interrupts add implementation complexity (resume logic, approval queue UI, careful thread/async boundary handling) but deliver precise control, automatic state replay, and first-class observability. For regulated enterprise domains the extra complexity is justified; for internal low-risk copilots we still offer a lighter async path.
These patterns—Redis-backed conversation memory with TTL, thread-safe singletons via double-checked locking, LangGraph interrupts for HITL, multi-layer guardrail middleware, and SSE streaming—have allowed us to run production agentic platforms at scale while meeting strict non-functional requirements around correctness, auditability, and responsiveness. The architecture continues to evolve, but the core lessons on state as a first-class citizen remain unchanged.
Leave a Reply