Customer Support Agent
Running Verification on a Customer Support Agent
Agent Implementation
The customer support agent is implemented using LangGraph as a multi-agent system where a supervisor agent is responsible for routing each customer request to the right agent (customer agent for direct response to customer, database coodinator for database transactions, and knowledge agent for a RAG query to the company knowledge base).
Defining the contract specification
This specification is made up of 4 scenarios: each containing the data for the testbot to simulate a scenario along with the requirements defined in contracts.
Run Simulation
Conversations are simulated using the Relari Testbot for each scenario. Check out more details in the Simulation section.
Verification Results
The following is the verification results post-execution. We can see that the con-shipping-1
and con-product-1
contract are satisfied while the con-refund-1
and con-refund-2
contracts are not.
The detailed verification results with reasoning are saved in the output/verify_4cf4b3f5.json
file.
────────────────────────────────────────────────────── Trace f3ec911405e760de9d9c64664c26ed6b ────────────────────────────────────────────────────── con-shipping-1 (SATISFIED)
┏━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Type ┃ Qualifier ┃ Requirement ┃ Satisfied ┃ ┡━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ PRE │ MUST │ Customer has provided order number order_01JGMMMPHCQM01FBAHY23FDA1 │ Yes │ │ PRE │ SHOULD │ Customer has provided email address for verification │ Yes │ │ PATH │ MUST │ Use email address to retrieve associated orders, and verify the order number doesn’t exist │ Yes │ │ PATH │ MUST │ Do NOT make up any shipping status, delivery dates, or any reason for the order not existing │ Yes │ │ PATH │ SHOULD │ Do NOT ask for unnecessary personal information beyond order number and email │ Yes │ │ POST │ MUST │ Inform customer that the order number does not exist in the system │ Yes │ │ POST │ SHOULD │ Provides escalation contact information to customer │ No │ └──────┴───────────┴──────────────────────────────────────────────────────────────────────────────────────────────┴───────────┘ ────────────────────────────────────────────────────── Trace 73b754a05920cc9ce5d26abbb73fe0d4 ────────────────────────────────────────────────────── con-refund-2_1 (UNSATISFIED)
┏━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Type ┃ Qualifier ┃ Requirement ┃ Satisfied ┃ ┡━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ PRE │ MUST │ Customer provides or selects order number order_01JGQF1YAYNFDV874MKFR33CAA │ Yes │ │ PRE │ SHOULD │ Customer provides email address if asked │ Yes │ │ PATH │ MUST │ Collect when the damage was first noticed, location of damage, and whether the item was worn │ Yes │ │ PATH │ MUST │ Check against the return policy and determine that the order is not eligible for a refund │ Yes │ │ POST │ MUST │ Deny the refund request │ No │ │ POST │ SHOULD │ Escalate to a manager if the customer insists on a refund │ No │ └──────┴───────────┴──────────────────────────────────────────────────────────────────────────────────────────────┴───────────┘ ────────────────────────────────────────────────────── Trace ed602aa95bbb251eb48e8f8ff56c8980 ────────────────────────────────────────────────────── con-refund-1 (UNSATISFIED)
┏━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Type ┃ Qualifier ┃ Requirement ┃ Satisfied ┃ ┡━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ PRE │ MUST │ Customer provides order number │ Yes │ │ PRE │ SHOULD │ Customer provides email address if asked │ No │ │ PRE │ SHOULD │ Customer should provided photo of damage to the sweater if asked │ No │ │ PATH │ MUST │ Collect when the damage was first noticed, location of damage, and the photo of damage │ No │ │ PATH │ MUST │ Check against the return policy and determine that the order is eligible for a refund │ Yes │ │ POST │ MUST │ Provide the refund id to the customer │ Yes │ │ POST │ MUST │ Process refund request as a store credit │ No │ └──────┴───────────┴────────────────────────────────────────────────────────────────────────────────────────┴───────────┘ ────────────────────────────────────────────────────── Trace 65d7b69d78be0a80638dd1304c20f0fa ────────────────────────────────────────────────────── con-product-1 (SATISFIED)
┏━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Type ┃ Qualifier ┃ Requirement ┃ Satisfied ┃ ┡━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ PRE │ MUST │ Customer express interest in shirts first (not t-shirts specifically) │ Yes │ │ PRE │ SHOULD │ Customer specifies that they want t-shirts and not sweatshirts │ Yes │ │ PATH │ MUST │ Ask customer for their preferences │ Yes │ │ PATH │ SHOULD │ Clarify if customer want t-shirts or sweatshirts │ No │ │ PATH │ MUST │ Do NOT recommend non-shirt products │ Yes │ │ PATH │ MUST │ Do NOT make up product features or specifications │ Yes │ │ POST │ MUST │ Provide a comprehensive list of available t-shirts │ Yes │ │ POST │ MUST │ Include key details like description, sizes, and available colors │ Yes │ │ POST │ SHOULD │ Provide photos of t-shirts │ Yes │ └──────┴───────────┴───────────────────────────────────────────────────────────────────────┴───────────┘
Example Certificate at Runtime
The following are two example runtime certificate received during the execution.