everyone this is Sean
let's think about how to design an AI agent system
like a pro it doesn't matter if you're technical or not
if you're a product manager data scientist or computer scientist does not matter
the goal here really is to understand how do we think holistically about
you know starting from the front end
all the way to the back end
to database and setting up AI agents
so that your app will be smarter than just a traditional Sass product
Problem to Solve Today
How to Design an AI Agent System for
E-Commerce DTC Brands to Manage Customer Support
- Goals: >=70% Automation, CSAT >4.5/5, p50<1s, p95<2.5s
- Scope: Returns, Exchanges, WISMO
- Rules:
- Return window (<30d only), must return in good conditions
- Refund Policy (<$50 auto, >=$50 manager)
- Exchange Policy: good conditions, categories of products
- Route to human: asked by user, emotion detection
- Backend/DB:
- Peak RPS & Seasonality
- Data Storage to Relational DB or Vector DB
today's example is about how do we design this agent system for the e-commerce
for the e-commerce direct to customer brands
DTC brands to manage their customer support
DTC brands to manage their customer support
so as we know that customer support is a field
that traditionally has a lot of human involvement
when it comes to things like return products
exchange products or you know
asking about the status of where my product is
when it's being shipped and um
for a lot of e-commerce company
they probably cannot afford to hire a bunch of like
call centers or message replying agents
so this would be a perfect use case for them to
you know increase their response rate as well as
improve the satisfaction for their customers
Goals
so now let's jump in and think about
how do we think through this step by step okay
so the first thing is that our goal today
is that we want to make sure that
we will have a
over 70% automation in this customer support flow
which means that the other 30% probably be like
we're gonna loop the humans back into
the conversation when a
e-commerce site is talking to a customer right
and also we wanna make sure that the customer satisfaction rate
CSAT is greater than 4.5 out of 5 right
and last but not least
we wanna make sure that the response
with the 50% of the percentage of the customers
is below 1 second and then 95% of your customers
will get a response under 2.5 seconds
why is this important this is important
because if the response time is too long
it really affects your satisfaction from our customers
okay so this is our goal
so let's also think about what is our scope today
Scope: Returns, Exchanges, WISMO
so let me just delete this flow
and let's walk through it together
so firstly let's think about the rules of this scope
Rules:
- Return window (<30d only), must return in good conditions
- Refund Policy (<$50 auto, >=$50 manager)
- Exchange Policy: good conditions, categories of products
- Route to human: asked by user, emotion detection
- Backend/DB:
- Peak RPS & Seasonality
- Data Storage to Relational DB or Vector DB
in order to return to the customer
we need to make sure that the products for example
will only be returned if it's bought over
under 30 days before
and it must be returned in good conditions right
otherwise the airlines probably will just allow any
products to be returned at any time
which is not what we want right
the second rule we think about is um
how do we think about the refund policy right
in this case most companies probably have a refund policies
for example if it's under $50
and if they're using a human call center as a service
then they will automatically let it to um
return the product but if it's over or equal to $50
then they probably need to loop in their manager
we want to mimic the exact user behavior here
the third one is called exchange policy
so if a customer wanna exchange the product
they also need to make sure a few things
um it needs to be in good conditions
and certain categories of products cannot be returned
because for example
if it's food and you already open it
you cannot really return it right
and lastly that um
how do we route back to a human right
there could be all sorts of scenarios here um
it could be because the user
directly asked that they want to talk to a human
or we have detected that there's some very negative
of emotions in the chat
so uh
we must involve a human being to
basically provide emotional support for the customer
okay
um from a back end and database perspective
there are a few things that we should consider
in this case let me just paste it right in
the first thing is that
when and how much is the peak for request per second
and about the seasonality as well
imagine this is Black Friday or Christmas time
then probably there will be a lot more people
doing online shopping
versus the other period of time of the year
and also we need to think about how we're gonna store
all the chat history um
information about the transactions
about return and logistics
do we store it in a relational database
or do we store it in a vector database
so here's a concept we need to introduce for AI agents
which is vector database
what it really means is that traditionally
we basically define data tables in
imagine you have an Excel sheet
there are rows and columns
every column means a different feature
every row means like a different entry right but
what it really means is that traditionally
we basically define data tables in
imagine you have an Excel sheet
there are rows and columns
every column means a different feature
every row means like a different entry right but
in order to search some of the say
unstructured data
like the return policy or refund policy
these kind of things are not
gonna be stored in traditional
like table format
they're probably a paragraph or PDF right
so the way to do it is that we can
store this information in a vector database
in which we will embed each word or each
you know paragraphs into uh vectors right
so what we mean by that is that
we're gonna turn these words into numbers
that are in high dimensions
if you're really not from a technical background
just think of it as like we gotta digitize something
like if you're looking at a photo on your iPhone
it will be stored in digits
and then when they're stored
in digits you'll be able to search them by calculating
some similarity between photos
similarly here
we're allowing it to calculate similarities between
vectors right
so given the current scope
the goal the rules the back end database
let's jump in to start designing the agent system here
the first thing we're gonna do is that
we need to think about
when will a user start to interact with our system
right
so in this case our user channels are gonna be website
chat and emails okay so
User Channels
- Web Chat, Email
Gateway
- Auth/SSO,
- PII,
- RateLimit,
- dedupe
Auth
User Auth, Fetch Chats/Emails
Relational DB
Auth
Ask a question:
I want to return my
product X.
Respond to the user
real-time
Human Approval
Router Agent
- understands
user intent
- decides agent
routing
Q&A Agent
- speak to user
all the time
Decide to take the
Return Route Action
Return the latest
status of return back
to Q&A Agent to talk
to the user
Return Planner
Agent
- decides
functional calling
- Or, a deterministic
workflow
Vector DB
(FAQs, Policies)
RAG: Check Policy Via
VectorDB
Align with Policy
Inform the agent
Policy/Keywords:
- Capability Tokens, RMAs,
human approach, read-
back(confirmation)
IF it fits the policy,
call Shopify API for
return
Shopify API
- track orders status, Return
Merchandise Auth (RMA),
exchanges
Stripe API -
Payments/Refund
CRM
- write/fetch deal
pipeline,
create/update
tickets
How The Router Agent Proceeds
So here we're gonna introduce a gateway
a gateway is going to deal with the authentication
single sign on or PII which is for privacy of the user
or you're gonna think through the rate limit
how many times people can actually use this product
request for this product and checking for duplication
so all these kind of things are sort of
we need to double check with the gateway
let's add a bit of arrow to confirm this relationship
before we continue to process any information
we need to finish authentication first
CRUD with Relational DB
and also in order to load the previous chat and emails
we need to query our data from a relational database
remember we talked about the rectangle data tables
so what happens here is basically user auth
fetch the chats and emails okay
How Would AI Agents Interact With User Chats
we're gonna start
think about how would the agent help with
talking to these conversations from customers okay
imagine a customer has just asked a question
ask a question they said
I want to return my
product X.
Respond to the user
real-time
Router Agent
- understands
user intent
- decides agent
routing
Q&A Agent
- speak to user
all the time
Decide to take the
Return Route Action
Return the latest
status of return back
to Q&A Agent to talk
to the user
Return Planner
Agent
- decides
functional calling
- Or, a deterministic
workflow
Vector DB
(FAQs, Policies)
RAG: Check Policy Via
VectorDB
Align with Policy
Inform the agent
Policy/Keywords:
- Capability Tokens, RMAs,
human approach, read-
back(confirmation)
IF it fits the policy,
call Shopify API for
return
Shopify API
- track orders status, Return
Merchandise Auth (RMA),
exchanges
Stripe API -
Payments/Refund
CRM
- write/fetch deal
pipeline,
create/update
tickets
so here for the router agent
let's think about the edge case first right
if the router agent realized that this customer is very angry
like the customers say that I have
I must deal with like or they ask a very
very complex question
and we think that our agent decides that it's
way
beyond the confidence that I have to deal with myself
we need to have a mechanism to allow it to trigger
looping the human back into this agentic system
so that the humans will take over
so that our overall customer satisfaction
Ray will not be affected right
so I'm just gonna add in this human in the loop
Human Approval
so now in this case the human will start to approve
or disapprove any request from the users
How The Router Agent Proceeds
if the router agent decides
now that this is a valid request
and let me just double check if it fits with our policy
so that I will process the rest of the steps for you
then we will allow you to move on to the next step
so the next step we're calling it a planner agent
let me just move it back here
Decide to take the
Return Route Action
Planner Agent
- decides
functional calling
- Or, a deterministic
workflow
if it fits
the policy
call Shopify API for
return
Shopify API
- track orders status, Return
Merchandise Auth (RMA),
exchanges
Stripe API -
Payments/Refund
Payment Refund done
after this the return planner agent
should be updating all the context
back to this Q&A agent because this Q&A
agent is really doing the conversation
with the original user channels
which includes the web chat or the emails right
The Q&A Agent is the CEO
so imagine this QA agent is just like the central brain
or the CEO of the company
who needs to do all the communication
with their customers and then the rest of these people
are just part of the organization
who's doing their task right
just that the router agent is sort of at a higher level
return planner is the one that the router agent
decides to route to
and then you could also route to Exchange Planner or
you know where is my order planner
all these kind of tools right
but after this planner has done its job
what you should do is that you should always update
the information or the context back to the Q&A agent
right so the Q&A agent is like OK
so um
return the latest updates
return the latest status of return back
to Q&A agent to talk
to the user
so this can also include situations when
the question or the product doesn't fit into the policy
for return in this case
we will also
update the latest information back to the Q&A agent
so they will be able to talk to um
the user right
respond to the user real-time
AI Agent System Design Overview
Relational DB
User Auth, Fetch Chats/Emails
Gateway
- Auth/SSO,
- PII,
- RateLimit,
- dedupe
Auth
User Channels
- Web Chat, Email
Human Approval
Router Agent
- understands
user intent
- decides agent
routing
Q&A Agent
- speak to user
all the time
Decide to take the
Return Route Action
Return the latest
status of return back
to Q&A Agent to talk
to the user
Return Planner
Agent
- decides
functional calling
- Or, a deterministic
workflow
Vector DB
(FAQs, Policies)
RAG: Check Policy Via
VectorDB
Align with Policy
Inform the agent
Policy/Keywords:
- Capability Tokens, RMAs,
human approach, read-
back(confirmation)
IF it fits the policy,
call Shopify API for
return
Shopify API
- track orders status, Return
Merchandise Auth (RMA),
exchanges
Stripe API -
Payments/Refund
CRM
- write/fetch deal
pipeline,
create/update
tickets
so we got a pretty solid
system right here for the goal we have right
so what do we still need to do
Observability, Metrics, Evals
few things No. 1 is that as an agentic system
we should always be thinking about observability
metrics and evaluations okay remember our goal
is that we need 70% are automation
30% are looping back to human right
and then the satisfaction rate should be over 4.5
and the response time should be um
this much right
so what we do is that we gotta track
the relevant metrics across the entire system
to make sure that these metrics are being met
these goals are being met
okay so that's the first thing
the second thing is that it depends on um
this current website or this current client
is already dealing with some external
like say CRM systems
perhaps there should be some automatic triggers as well
for us to write back to the CRMs
okay so we could add something like this here
a CRM system that will fetch
write/fetch deal
pipeline,
create/update
tickets
CRM Integration
Observability, Metrics, Evals
User Auth, Fetch Chats/Emails
Relational DB
Gateway
- Auth/SSO,
- PII,
- RateLimit,
- dedupe
Auth
User Channels
- Web Chat, Email
Human Approval
Router Agent
- understands
user intent
- decides agent
routing
Q&A Agent
- speak to user
all the time
Decide to take the
Return Route Action
Return the latest
status of return back
to Q&A Agent to talk
to the user
Return Planner
Agent
- decides
functional calling
- Or, a deterministic
workflow
Vector DB
(FAQs, Policies)
RAG: Check Policy Via
VectorDB
Align with Policy
Inform the agent
Policy/Keywords:
- Capability Tokens, RMAs,
human approach, read-
back(confirmation)
IF it fits the policy,
call Shopify API for
return
Shopify API
- track orders status, Return
Merchandise Auth (RMA),
exchanges
Stripe API -
Payments/Refund
CRM
- write/fetch deal
pipeline,
create/update
tickets
so again this is a very simple
overview on how do you think about the agentic system
um from a non technical background
from a non technical perspective
but also I know that I have introduced
quite a lot of technical concepts here too
but um if you are technical
you realize that
there are a lot of things that we're missing out right
there are a lot of things that we didn't mention
for example about scalability about
things that are related to like the request per second
seasonality all these server side of things
I think at the end of the day
if you're a product manager
you would need to discuss with your engineering leader
anyways to figure out these things out
this basic concept of what we need
in order to set up this product
ready for production
I hope this is helpful I hope
you have learned something from this
and let me know if you like this kind of format of video
I'm happy to make more
and I'll probably also be making
some videos to explain
how do you actually build a system like this with code
so stay tuned if you like this video
like and subscribe and make a comment down below
thanks very much cheers