What You Will Learn

• Architect a multi-tenant B2B AI agent platform with microservices and an API gateway
• Implement secure auth, OTP verification, JWT sessions, gRPC, and RabbitMQ-driven email flows
• Build an agent engine with memory agent, worker agents, delegation, and human-in-the-loop handoff
• Design a three-tier memory system: working logs, long-term (MongoDB + PGVector), and Redis caching with compression
• Ingest business documents for RAG, implement PGVector retrieval, and capture/update leads during conversations

Study Guide

Introduction: Why This Course Exists and What You'll Actually Build

Most people think building an AI agent means calling an LLM API and wrapping it in a chat window. That's not a product. That's a demo. If you want to build something that enterprises can actually use , something that handles real customers, persists memory across conversations, retrieves company knowledge, captures leads, and escalates to humans when needed , you need to think like a systems engineer, not just a prompt engineer.

This course is about building a complete, production-ready B2B AI agent platform. Not a toy. Not a tutorial that stops at "hello world." We're talking about a multi-tenant system where businesses can create their own conversational agents, embed them into their websites, and manage customer interactions at scale. You'll learn how to architect this with microservices, how to make services talk to each other with gRPC, how to handle asynchronous events with RabbitMQ, and how to build a sophisticated memory system using LangChain and LangGraph.

Here's the thing though. The value isn't in any single technology. It's in how they all fit together. You'll see why an API gateway is your only public entry point. Why email verification shouldn't block your registration flow. Why your agent needs three different types of memory. Why observability is a feature, not an afterthought. When you finish this course, you'll have a mental model for building AI systems that survive contact with real users.

Section 1: The Architecture , Why Microservices Beat Monoliths for AI Systems

Let's start with the big picture. The platform we're building is a B2B AI agent builder. Think of it as a service that lets companies create conversational agents for sales, customer support, lead qualification, and marketing. These agents live on company websites, answer product questions, qualify prospects, and hand off complex conversations to human operators.

Now, you could build this as a monolithic application. One big codebase. One deployment unit. But here's the problem with that approach: if one component fails , say the email notification system , the entire application goes down. Every user, every agent, every conversation. That's what we call a single point of failure, and for a B2B platform, that's unacceptable. Your customers' businesses depend on your system. If you go down, they lose revenue.

Microservices architecture solves this by isolating failures.
Each service is independently deployable and loosely coupled. If the notification service crashes, the agent service keeps running. If the auth service needs to restart for maintenance, users can still chat with their agents. This resilience is the core reason we're splitting the system into dedicated services.

Let me give you a concrete example. Imagine a customer is mid-conversation with a support agent, and suddenly the notification service dies because the email provider is rate-limiting requests. In a monolith, that conversation dies too. The customer gets an error. They leave. In our microservices architecture, the conversation continues uninterrupted. The notification service can fail, restart, or even be redeployed without affecting anything else.

Another example: scaling. Let's say one customer's agent becomes incredibly popular. Traffic spikes. In a monolith, you'd have to scale the entire application , including the auth service, the notification service, everything , just to handle the load on one feature. With microservices, you can scale just the task service that runs the agents. Spin up three more instances. The other services don't care. This is the kind of flexibility that makes microservices worth the complexity.

So what are the actual services we're building? Here's the breakdown. You've got the API gateway, which is the single public entry point for all client requests. It handles routing, authentication, and request validation. Then there's the auth service, which manages user registration, OTP verification, login, and token generation. The task service is the heart of the platform , it hosts the agent creation logic, graph-based orchestration, memory systems, knowledge-base pipelines, and lead-capture tools. And finally, the notification service consumes events and sends transactional emails.

We're also running infrastructure services. RabbitMQ handles asynchronous communication between services. MongoDB stores user data, chat history, working memory, long-term memory, and agent configurations. Redis caches working memory and short-term memory for fast access. And PGVector , a PostgreSQL extension , serves as our vector database for embedding memory summaries and organizational knowledge.

Now, here's a critical point about how these services communicate. For internal service-to-service calls, we're using gRPC with Protocol Buffers. Why not REST? Because gRPC is faster, strongly typed, and better suited to high-throughput internal traffic. Each service owns a `.proto` contract that defines its methods and message structures. This makes service boundaries explicit. You can't accidentally break another service's interface because the contract is defined in code and compiled. If you change a method signature, the compilation fails. That's a good thing.

But not everything should be synchronous. Asynchronous tasks , like sending verification emails after user registration , are handled through RabbitMQ. Services publish events to named queues, and consumers process them independently. This decoupling means the auth service doesn't need to wait for the email service to complete. The user gets their response immediately, and the email gets sent whenever the notification service gets around to it. If the email service is slow, that's fine. The user isn't blocked.

Containerization ties it all together.
The entire platform runs as Docker containers orchestrated with Docker Compose. Each service has its own Dockerfile, and the compose file defines ports, volumes, networks, and dependencies. The reference setup runs nine containers, including databases and infrastructure services. There's a shared bridge network so containers can talk to each other by service name , `auth-service`, `task-service`, `pgvector-db` , rather than by IP addresses that might change.

Here's a production practice that matters more than you might think: minimizing image size. If you're not careful, your development images can balloon to roughly 2.2 GB per service. That's insane. A production-ready approach uses slim runtime images to reduce deployment size to approximately 500 MB per service. That's a four-fold reduction. It means faster deployments, less disk usage, and quicker cold starts. When you're running nine containers, this adds up fast.

Section 2: Authentication and Security , The Gateway as Your Only Public Face

Security in a microservices architecture isn't just about passwords and tokens. It's about defining your attack surface and shrinking it. The principle here is simple: the API gateway is the only service whose port is exposed publicly. Everything else , MongoDB, RabbitMQ, Redis, PGVector, the internal services themselves , lives on the Docker network, unreachable from the outside world.

Let me give you a concrete example of why this matters. Imagine you expose the MongoDB port directly to the internet. Now anyone who discovers that port can try to connect. Maybe they brute-force the password. Maybe there's a misconfiguration. Suddenly your entire database is compromised. But if MongoDB is only accessible within the Docker network, an external attacker can't even reach it. They'd have to go through the gateway first, which means they'd have to get past your authentication, your validation, and your routing logic. That's the security boundary we're building.

Now let's walk through the user lifecycle.
Registration starts when a user submits an email and password through the API gateway. The gateway validates the input using Zod schemas , this is our first line of defense. Then it forwards the data via gRPC to the auth service. The auth service validates again, hashes the password using bcrypt, and creates a user record in MongoDB with an `isValidEmail` flag set to false. Here's where the event-driven magic happens: the auth service generates an OTP code and publishes a `user.created` event to RabbitMQ. The notification service consumes this event and sends the OTP through NodeMailer. The user receives their code, submits it to the gateway, and the auth service marks their email as verified.

Why RabbitMQ here? Let's say the email provider is down. Without the message broker, the registration request would hang or fail. With RabbitMQ, the event sits in the queue. The user gets a "check your email" response immediately. When the email service comes back online, it processes the queued events. The user experience is never degraded by infrastructure issues. That's resilience in practice.

For testing this flow, you can use Mailtrap , a sandbox SMTP service that captures emails instead of delivering them. It's perfect for development because you can inspect the OTP email without actually sending anything to a real inbox. When you're ready for production, you swap in your real email provider credentials.

Login and token strategy follow standard JWT patterns.
Authenticated users receive an access token and a refresh token. These are signed with private keys configured in environment variables. The access token is short-lived and used to access protected endpoints. The refresh token is long-lived and allows users to maintain sessions without re-entering their credentials. The API gateway includes middleware that verifies tokens before forwarding requests to internal services. This creates a centralized authorization boundary , no service needs to implement its own token verification because the gateway handles it.

Here's a best practice worth internalizing: the gateway should be responsible for authentication, request validation, and rate limiting. When you centralize these concerns, you avoid the trap of each service implementing security differently. One service might forget to validate a field. Another might use a weaker hashing algorithm. The gateway ensures consistency across the board.

Section 3: The Agent Engine , Memory Agent vs. Worker Agents

This is where the platform gets interesting. The agent system has two main layers. The memory agent is the orchestrator. It maintains conversation memory, uses tools, and decides when to pass control to a worker agent. The worker agent is a task-specific agent created by the platform user. Examples include B2B website conversational agents for sales, lead qualification, customer support, and marketing.

Here's a key distinction: the memory agent is not designed for general-purpose work. It's not going to write code for you or compose poetry. It operates as an administrative coordinator. It manages context, decides what to store and retrieve, and routes requests to the appropriate worker. Think of it as the dispatcher in a call center. It doesn't handle the customer directly , it figures out which specialist should.

The memory agent's responsibilities are clear.
It receives all user input first. It assembles context from working memory and long-term memory. Then it decides how to respond. If the user asks something that requires business-specific knowledge , say, pricing for a SaaS product , the memory agent invokes the delegate agent tool with a message containing the user input, relevant chat history, and memory context. A routing function detects the transfer keyword in the memory agent's output and routes the graph to the worker agent node. The worker agent then processes the request, generates a response, and returns it to the user.

Let me give you a concrete example. A customer visits a company's website and types, "Hello, I want to contact your manager." The memory agent receives this. It recognizes that this is a sales-related request, not a memory operation. It invokes the delegate agent tool. The routing function sees the transfer keyword and moves the conversation to the worker agent , say, the "Jarvis Sales Agent." The worker agent, with its sales persona and company context, responds professionally: "I'd be happy to help you with that. Could you share a few details about your company and what you're looking for?"

Worker agents are created by platform users through a simple configuration model.
The user provides an agent name, a persona describing the tone and style, a goal defining the primary objective, a category for domain classification, and company context with background information. This data is stored in MongoDB and retrieved at runtime to generate the worker agent's system prompt.

The system prompt is constructed from structured tags that the generation logic inserts into a prompt template. The final prompt typically includes the company identity and role, product or service category, company background, stated goals and objectives, desired tone and communication style, and strict boundary rules. For example, a sales agent might be instructed to refuse code generation, mathematical computation, or topics outside company scope. These boundaries make worker agents reliable for narrow, business-specific conversational tasks.

Here's a concrete example of what that prompt structure looks like in practice. The agent name is "Jarvis Sales Agent." The persona is "professional, friendly, solutions-oriented." The goal is "qualify inbound enterprise leads and schedule demos." The category is "sales." The company context describes a SaaS company selling project management software. The prompt tells the agent to stay in scope, use the knowledge base for pricing questions, and capture leads when appropriate. It also explicitly says: "Do not provide coding assistance. Do not perform complex calculations. If asked, politely redirect to the sales conversation."

Section 4: The Memory Hierarchy , Working, Long-Term, and Short-Term

Most AI applications treat memory as an afterthought. They stuff the entire conversation history into the context window and hope for the best. That works for short demos, but it falls apart in production. Conversations grow. Context windows fill up. Costs skyrocket. And the model starts losing track of important details from earlier in the conversation.

This platform implements three forms of memory, each serving a different purpose.

Working memory tracks ongoing interactions for the current thread and day.
It's stored in MongoDB, structured as date-based daily logs. Each log grows as users exchange messages with the agent. When you need to present this to the LLM, you convert it to markdown format , a daily log with entries labeled "User:" and "AI:". This gives the model a clean chronological view of the conversation.

Here's the critical part: working memory can't grow forever. When it exceeds a configured token threshold , say, 15,000 tokens , the platform triggers a compression and archival process. First, it extracts the full working-memory log. Then it sends the log to an LLM with a structured compression prompt. The LLM produces a concise summary of 1,000 to 2,000 tokens. This summary is embedded into PGVector for semantic retrieval. The raw conversation log is appended to an archive collection. And the active working-memory collection is cleared for the current thread.

Let me give you a concrete example. A customer has been chatting with a support agent for an hour. They've discussed three different issues, shared their company size, mentioned their budget, and asked about integration options. That's easily 15,000 tokens of conversation. Without compression, the next message would push the context window over the limit. With compression, the system extracts the key facts , "Company size: 50 employees. Budget: $10k/year. Interested in Slack integration. Issue #1 resolved. Issue #2 escalated." , and stores that summary for future retrieval. The raw log goes to the archive for deep historical searches.

Long-term memory stores stable user facts, preferences, goals, and personal details.
This is the memory that persists across conversations. It's stored twice: as structured rows in MongoDB, and as compressed summaries embedded into PGVector. The MongoDB rows provide structure and filtering , you can query for all memories of a certain category or importance level. The PGVector embeddings enable semantic retrieval , you can search for "what did this user say about their budget?" and find the relevant memory even if the exact words don't match.

Why store it twice? Because different queries need different access patterns. A structured query , "get all memories with importance = critical" , is fast and easy in MongoDB. A semantic query , "find memories related to budget discussions" , requires vector similarity search. Having both representations means you can handle both types of queries efficiently.

Short-term memory is cached in Redis for fast access.
When the memory agent needs to retrieve long-term memory, it first checks Redis using a stable cache key like `long_term_memory:${userId}:${agentId}`. If the cache hits, great , no database query needed. If it misses, the system retrieves from PGVector and populates the cache. When the `write_memory` tool is called, the system invalidates the Redis cache so the next retrieval gets fresh data.

The memory agent is equipped with explicit tools.
The write memory tool stores relevant information into long-term memory. The search memory tool performs semantic retrieval through PGVector. The BM25 archive retriever performs keyword search over archived working-memory logs. And the delegate agent tool transfers control to a worker agent. These tools are exposed to the LLM through structured schemas, making memory operations explicit and observable. The LLM doesn't just "remember" things implicitly , it actively decides when to store and retrieve information.

Here's an example of how this plays out. A customer says, "I'm the CTO of Acme Corp, and we're looking for a solution that integrates with Salesforce." The memory agent recognizes this as a stable fact worth remembering. It calls the write memory tool with the content, the category "fact," and importance "high." The tool writes the memory to MongoDB, invalidates the Redis cache, and schedules an embedding job via Agenda , our background job scheduler. Next time the customer chats, the memory agent's search memory tool retrieves this fact and the agent can reference it: "Last time we spoke, you mentioned Acme Corp was interested in Salesforce integration. Have you had a chance to evaluate that?"

Section 5: The Knowledge Base System , RAG for Business Documents

An AI agent is only as useful as the information it can access. A sales agent that doesn't know your pricing is useless. A support agent that doesn't know your return policy is a liability. That's why the platform includes a knowledge base system that allows users to upload organizational documents , FAQs, product documentation, pricing guides, company policies , that agents can consult during conversations.

The ingestion workflow is straightforward.
A user uploads a PDF through the API gateway. The file is temporarily held in memory , using multer with in-memory storage , and forwarded to the task service via gRPC. The task service saves the file temporarily to disk, extracts the text using a PDF document loader, and passes the documents to the knowledge base embedding pipeline. The pipeline splits the text into child chunks and parent documents using a parent-document retriever strategy. Both representations are embedded into PGVector with metadata tags , userId, agentId, docType, fileName, uploader. The original file is stored on disk for reference.

Why the parent-child pattern? Here's the problem it solves. Small chunks are great for retrieval because they're precise , you find the exact passage that matches the query. But small chunks lack context , they might not contain the full answer. The parent-child approach gives you both. You retrieve the small child chunk that matches the query, then use its parent ID to recover the complete parent passage that contains the full context. It's a best of both worlds approach.

Retrieval uses a multi-stage pipeline.
When a worker agent calls the search knowledge base tool, the system runs a vector similarity search in PGVector for documents linked to that agent and user. It converts the retrieved documents to a formatted string. Then it passes the documents and the query to a custom LLM extractor that removes irrelevant sections, de-duplicates repetitive information, and outputs only the minimal relevant context. The tool returns an annotated string wrapped in knowledge base data tags.

Let me walk through a concrete example. A customer asks, "What's the price of your Enterprise plan?" The worker agent recognizes this as a business-specific question and calls the search knowledge base tool. The system searches for documents related to pricing. It finds several chunks , some about the Enterprise plan, some about the Pro plan, some about a completely different topic that happens to contain the word "price." The LLM extractor filters out the irrelevant chunks and returns only the pricing information for the Enterprise plan. The worker agent then formulates a response: "Our Enterprise plan is $499 per month, billed annually. It includes unlimited projects, priority support, and custom integrations."

The worker agent's system prompt instructs it to use the knowledge base tool whenever a question depends on company-specific information.
It also instructs the agent to honor boundaries and not hallucinate pricing or terms if no knowledge base data is available. If the agent doesn't know the answer, it should say so and offer to connect the customer with a human representative.

Section 6: Lead Management , Turning Conversations into Revenue

For a B2B platform, lead capture isn't a nice-to-have. It's the whole point. Every conversation is an opportunity to collect contact information, qualify prospects, and route them to the sales team. The platform models leads as customers with first name, last name, email, and optional additional fields.

Two tools are provided to worker agents.
The capture lead tool accepts lead information and creates a new customer record. The update lead tool modifies existing lead details, including email and contact preferences. These tools allow conversational agents to naturally collect contact information, qualify prospects, and route valuable leads to human sales teams.

Here's how it works in practice. A customer says, "I'm interested in learning more about your product." The worker agent, following its system prompt, engages in a bit of qualification first: "I'd be happy to help. Could you share your name and email so our sales team can follow up with you?" The customer provides the details. The agent calls the capture lead tool with the first name, last name, and email. The tool checks if a lead with that email already exists. If not, it creates a new customer record in MongoDB. If it does exist, it might update the record with any new information.

Here's another example. A lead changes their email address. They tell the agent, "Actually, can you update my contact info? It's now john.doe@newwork.com." The agent calls the update lead tool with the old email and the new email. The tool finds the lead by the old email and updates the record. The agent confirms: "I've updated your contact information. Is there anything else I can help you with?"

Best practice: agents should confirm with the user before capturing or updating personal data.
The system prompt should instruct the agent to explain what will happen with the user's data. This builds trust and reduces the chance of users feeling their information was collected without consent.

Section 7: Human-in-the-Loop , The Escape Hatch That Builds Trust

No matter how good your AI agent is, there will be situations that require human judgment. A customer is angry and demands to speak to a manager. A complex technical question is beyond the agent's knowledge base. A high-value prospect needs a personal touch. For these situations, the platform includes a human takeover capability.

The interface provides a live conversation pane showing AI-customer interaction.
An admin can view the ongoing chat, see what the AI has said, and decide when to intervene. There's a button for human takeover. When clicked, the system switches the conversation from AI-driven to human-driven. WebSocket-based messaging pushes operator messages to the customer. The operator can have a real-time conversation. And when the situation is resolved, the operator can return control back to the AI agent.

Here's a concrete scenario. A customer is frustrated. The AI agent has tried three times to resolve their issue, but the customer keeps saying, "This isn't working. I want to talk to a real person." The AI, following its system prompt, recognizes the limit of its capabilities and suggests escalation. The admin sees this in the monitoring interface and clicks the takeover button. Now the admin can type messages directly to the customer. The customer sees the messages arrive in real-time via WebSocket. The admin resolves the issue, then clicks "return to AI" to hand the conversation back to the agent.

This feature is critical for B2B trust.
Customers need to know there's a path to a human when the situation requires it. Without this escape hatch, they'll lose confidence in the system and the company behind it. The chat history model includes fields for human loop metadata, making it possible to track when a human took over, what they said, and when control was returned.

Section 8: Observability and the Harness Tuning Loop

You can't improve what you can't see. That's the core principle behind observability in this platform. The system integrates LangSmith tracing to record every input, tool call, response, and latency metric. Observable traces show which memory tools were invoked and what arguments were passed, whether the memory agent delegated to the worker agent, which knowledge-base content was retrieved, the final response generated by the worker agent, and end-to-end latency for each interaction.

Let me give you an example of why this matters.
Your agent is answering customer questions, but you notice it's not using the knowledge base. Customers are asking pricing questions, and the agent is giving generic responses. Without tracing, you'd have no idea why. With LangSmith, you can see the exact tool calls , or lack thereof. You see that the agent's system prompt doesn't mention the knowledge base tool. You add a line to the prompt: "When asked about pricing, features, or policies, use the search knowledge base tool." Problem solved.

Here's another example. The memory agent is supposed to write important facts to long-term memory, but it never does. You check the traces and see that the write memory tool is available but the agent never calls it. The tool description is too vague. You update the description to: "Use this tool when the user shares personal information, company details, preferences, or goals that should be remembered for future conversations." Now the agent starts using it.

The platform embeds a clear improvement methodology: evaluate, observe, diagnose, engineer, repeat.
You send test inputs to the agent. You inspect outputs and trace data. You identify prompt weaknesses, routing errors, or tool mistakes. You modify system prompts, tool descriptions, or logic. And then you repeat the process. This is what the authors mean by "tune the harness, not the model." The quality of an AI application depends more on the surrounding system , prompts, tools, memory, retrieval , than on swapping foundation models.

Here's the practical implication. If your agent is performing poorly, don't immediately switch to a bigger model. First, look at your harness. Is the system prompt clear? Are the tool descriptions effective? Is the memory retrieval returning relevant context? Is the knowledge base properly indexed? In most cases, the problem isn't the model , it's the system around it.

Section 9: Deployment and the Demo Workflow

All of this runs in Docker containers. Each Node service has its own Dockerfile , using Node 20+ as the base image, copying package.json, running npm install, copying source code, and exposing the appropriate port. The docker-compose.yml file defines all services, their dependencies, networks, volumes, and environment variables. A shared bridge network allows containers to communicate by service name. Only the gateway port is exposed publicly.

The reference implementation validates core functionality through realistic scenarios.
Registration and email verification: a user signs up, receives an OTP, and validates their account. Agent creation: a user creates a "Jarvis Sales Agent" with a goal of qualifying inbound enterprise leads. Conversational interaction: a customer says, "Hello, I want to contact your manager," and the agent captures the request. Memory persistence: the customer states their name and a business detail, and the memory agent stores the facts in long-term memory. Delegation: the memory agent passes context to the worker agent, which delivers a professional response. Knowledge base retrieval: a user uploads a FAQ PDF, then asks a question that requires the worker agent to search the knowledge base. Lead capture: during conversation, the agent asks for contact details and creates a customer record. Human handoff: an operator takes over the chat, messages the customer, and returns control to the AI agent.

Here's a tip for testing: use production-like files and scenarios. Don't test with a one-page PDF when your customers will upload 50-page manuals. Don't test with a single conversation when your agents will handle hundreds of concurrent sessions. Test the way you'll operate in production.

Section 10: Key Design Principles and Technical Benchmarks

Let me distill the key insights from this architecture. Microservices improve resilience but introduce complexity. The platform avoids single points of failure by isolating services, but every service boundary requires disciplined contracts, message handling, and deployment automation. The API gateway is the only safe public entry point. Exposing internal service ports is a serious security risk. Memory is not monolithic. Effective conversational agents require separate working memory, long-term memory, cached short-term memory, and archival mechanisms. Context windows must be actively managed. Without summarization and compression, conversation threads grow indefinitely, degrade model performance, and increase cost. Hybrid retrieval improves answer quality. Combining vector search with keyword-based BM25 retrieval and LLM-based filtering provides both semantic understanding and precise recall. Subagents should be narrow and well-scoped. Worker agents perform best when given a clear goal, company context, persona, and explicit boundaries. Observability is a necessary production feature. Human handoff capability is essential for B2B trust.

Here are the technical constants to keep in mind.
Development Docker images run about 2.2 GB per service. Production targets are around 500 MB per service. The working memory compression threshold is configurable, tested at 15,000 tokens. Summaries after compression run 1,000 to 2,000 tokens. Long-term memory retrieval is limited to 50 rows per user and agent context. Internal gRPC ports map in the 5051-5053 range. The API gateway runs on public port 3000. Email testing uses Mailtrap. These values are a starting point , you'll tune them based on your model's context window, cost constraints, and application behavior.

Section 11: Implementation Roadmap , Where to Start

If you're building something similar, here's the phased approach I recommend. First, implement authentication, notification, and API gateway infrastructure. Get the registration flow working end-to-end , user signs up, receives OTP, verifies email, logs in. This gives you the foundation for everything else. Second, introduce the agent graph and memory systems. Build the memory agent with its tools , write memory, search memory, delegate agent. Get the basic conversation flow working. Third, add knowledge bases and lead tools. Upload documents, build the retrieval pipeline, implement lead capture and update. Finally, add observability, human handoff, and production hardening.

Here's the thing about building AI systems: the infrastructure matters as much as the intelligence.
You can have the best model in the world, but if your authentication is weak, your services are coupled, and your memory is a mess, you don't have a product. You have a liability. The architecture we've covered in this course is what separates a demo from a deployable system.

Conclusion: What You've Learned and What to Do Next

This platform demonstrates how modern AI systems can be engineered for serious enterprise use. It connects user-friendly agent creation with powerful backend components: distributed services, asynchronous event processing, persistent memory, knowledge retrieval, lead management, and human-in-the-loop controls. The architecture balances innovation and reliability by treating AI agents as part of a broader software system rather than as isolated models.

The most significant lesson is that production-ready conversational AI depends on the surrounding infrastructure. The memory hierarchy, retrieval pipelines, delegation logic, security boundaries, and observability layer are what transform a generic model into a trustworthy business tool. You can't just drop an LLM into a chat window and call it an agent. You need to think about how it remembers, how it learns, how it accesses knowledge, how it stays secure, and how it escalates when it's out of its depth.

Here's what I want you to do next. Don't just read this and move on. Build. Start with the authentication flow. Get the gateway talking to the auth service via gRPC. Add RabbitMQ and the notification service. Then build the memory agent with LangGraph. Give it the write memory and search memory tools. Add a worker agent. Upload a PDF and build the knowledge base retrieval. Integrate LangSmith and trace your first conversation. Then look at the traces and improve your prompts. This is the loop , build, observe, diagnose, improve. It's the same loop you'll use throughout your career building AI systems.

You now have the blueprint for a complete, production-ready B2B AI agent platform. The architecture is sound. The patterns are proven. The rest is execution. Go build something that matters.

Frequently Asked Questions

Introduction

This FAQ collects the practical questions that surface when building a production-ready AI agent platform with microservices, LangChain, LangGraph, and TypeScript. The answers cover architecture decisions, service communication, memory design, RAG implementation, and operational concerns. Each response reflects real-world experience running these systems , the kind of knowledge you gain from debugging production issues, not just reading documentation.

Architecture & Core Concepts

What is the overall system architecture of this B2B agent builder platform?

The platform uses a microservices architecture composed of four primary backend services plus infrastructure components:

Core services:
1. API Gateway Service - The entry point for all client requests. It handles authentication middleware, routes requests to appropriate services, and protects internal services from direct external access.
2. Auth Service - Manages user registration, email verification via OTP codes, and user authentication. It publishes events to a message broker for sending verification emails.
3. Task Service (Agent Service) - The core service containing all AI agent logic, including memory agents, worker agents, knowledge base systems, and lead management capabilities.
4. Notification Service - Consumes events from the message broker and sends emails (verification codes, notifications) using services like Mailtrap.

Supporting infrastructure:
- MongoDB - Primary database for storing user data, chat histories, working memory, and long-term memory
- PostgreSQL with PGVector - Vector database for embeddings storage and semantic search
- RabbitMQ - Message broker for asynchronous service communication
- Redis - Caching layer for long-term memory and working memory

All services run in Docker containers orchestrated with Docker Compose, communicating with each other via gRPC.

Why use microservices architecture instead of a monolith for this platform?

Microservices architecture offers two critical advantages for this type of platform:

Independent Deployability: Each service can be deployed, updated, or scaled independently without affecting other services. If the notification service fails, the auth service and agent service continue functioning.

No Single Point of Failure: Unlike monolithic architecture where a bug in one module can bring down the entire application, microservices isolate failures. A failure in the lead qualification agent does not affect the sales agent or the authentication service.

That said, microservices add operational complexity. You need Docker orchestration, service discovery, and distributed tracing. For a small team, a modular monolith might be simpler. The trade-off becomes worthwhile when you need independent scaling , for example, when the Task Service handles heavy AI workloads while the Auth Service stays relatively idle.

What is the role of the API Gateway in this system?

The API Gateway serves as the single entry point and security boundary for all external requests. It serves several critical functions:

Request Routing: Directs incoming REST API requests to the appropriate backend service (auth service, task service, etc.)

Authentication: Validates JWT tokens on protected routes before forwarding requests to backend services

Security: Only the gateway's port (3000) is exposed to the outside world. All internal service ports are hidden within the Docker network, preventing unauthorized direct access to services like MongoDB, RabbitMQ, or the individual microservices.

For example, when a user wants to create an agent, the request flows: Client → API Gateway → Task Service via gRPC.

How do you migrate from a monolith to this microservices architecture?

Start by identifying the boundaries between distinct business capabilities. In this platform, the natural splits are auth, agent logic, and notifications. Each becomes a separate service.

Migration steps:
1. Extract the auth flow first , it has the fewest dependencies on other modules.
2. Move notification sending into its own service, connected via RabbitMQ events rather than direct function calls.
3. Extract the agent logic last, since it depends on the most infrastructure (MongoDB, PGVector, Redis).
4. Introduce the API Gateway early to keep the external API contract stable while you refactor internally.

The key is to keep the external API stable during migration. Clients shouldn't notice any change. Use the gateway as a facade that routes to either the monolith or the new microservices during the transition period.

How do you scale the Task service horizontally when it becomes a bottleneck?

The Task Service is the most compute-intensive component because it handles LLM calls, embeddings, and vector searches. When it becomes a bottleneck, you have several options:

Scale replicas: Run multiple instances of the Task Service behind a load balancer. The gRPC client in the gateway needs to support multiple endpoints or use a service discovery mechanism.

Scale databases separately: MongoDB can be sharded, PGVector can use read replicas, and Redis can run in cluster mode.

Offload heavy work: Move embedding generation and memory compression to background jobs using Agenda. This keeps the request path fast.

Consider Kubernetes: Docker Compose works well for single-host deployments. For horizontal scaling, Kubernetes provides auto-scaling, rolling deployments, and built-in service discovery.

The gateway needs no changes if you use a load balancer or Kubernetes Service in front of the Task Service replicas.

Service Communication

How do microservices communicate with each other in this platform?

Services communicate using gRPC (Google Remote Procedure Call), which uses Protocol Buffers (protobuf) for defining API contracts and binary serialization for efficient data transfer. gRPC is preferred over REST for service-to-service communication because it's significantly faster and supports streaming responses.

The communication pattern involves:
1. Defining the API contract in .proto files (e.g., task.proto, agent.proto, chat.proto)
2. Loading these proto files in both the server and client implementations
3. Using generated gRPC clients to invoke methods on remote services

For example, the API Gateway loads task.proto and creates a gRPC client that connects to the Task Service running on port 5051 (internal Docker network address: task_service:5051).

What is the purpose of RabbitMQ in this architecture?

RabbitMQ serves as a message broker for asynchronous communication between services, particularly for event-driven workflows. In this platform, it's primarily used for:

User Registration Flow: When a user registers, the Auth Service publishes a "user_created" event to RabbitMQ. The Notification Service consumes this event and sends a verification email containing an OTP code.

Decoupling Services: Services don't need to be aware of each other. The Auth Service publishes events to a queue without needing to know how or when the notification will be sent.

The flow is: Auth Service → publishes event to RabbitMQ queue → Notification Service consumes event → sends verification email via NodeMailer.

This pattern means the Auth Service never blocks waiting for email delivery. If the Notification Service is down, the Auth Service still completes registration. When the Notification Service comes back online, it processes the queued events.

Why use gRPC instead of REST API for internal service communication?

gRPC offers several significant advantages for internal microservice communication:

Performance: Uses binary serialization (Protocol Buffers) instead of JSON, making it substantially faster with smaller payloads

Streaming Support: Native support for bidirectional streaming, essential for real-time AI agent responses

Strong Typing: Proto files define strict contracts, reducing integration errors

Code Generation: Generates client and server code from proto files, eliminating boilerplate

REST could technically be used, but gRPC is the better choice for high-performance, real-time AI applications where streaming responses are required.

What are common gRPC pitfalls when working with Node.js and TypeScript?

Several issues tend to trip up developers new to gRPC in Node.js:

Proto file path issues: Proto files must be accessible at runtime. In Docker, mount the proto directory as a volume or use absolute paths like /app/proto/task.proto.

Package name mismatches: The package declaration in your proto file must match what you reference in your gRPC client code. A mismatch causes cryptic "method not found" errors.

Async handling: gRPC callbacks in Node.js don't automatically handle promises. Wrap async operations in try/catch and call the callback explicitly.

Stream lifecycle: For streaming responses, you must properly handle the 'data', 'end', and 'error' events on the client side. Missing error handlers cause unhandled exceptions that crash the process.

Use TypeScript types generated from proto files to catch contract mismatches at compile time rather than runtime.

How does end-to-end chat streaming work from client to agent?

The streaming flow involves several layers working together:

Client → Gateway: The client sends a message via HTTP POST to the gateway's chat endpoint. The gateway immediately returns a text/event-stream response, keeping the HTTP connection open.

Gateway → Task Service: The gateway opens a gRPC streaming call to the Task Service's Chat method. The Task Service processes the message through LangGraph and streams response chunks back over gRPC.

Task Service → Gateway → Client: Each chunk received via gRPC is forwarded by the gateway as a Server-Sent Event (SSE) to the client. The client renders tokens as they arrive.

This layered approach means the client sees tokens appear in real time, which creates a much better user experience than waiting for a complete response. The streaming also lets you show intermediate states, like when the agent is calling a tool.

AI Agent System