You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"Prompt": "I have received this message that I owe $300 and I was instructed to pay the bill online. I already paid this amount and I am wondering why I received this message.",
609
+
"Completion": "report_payment_issue"
610
+
},
611
+
{
612
+
"Prompt": "I will not be able to attend the presentation and would like to cancel my rsvp.",
613
+
"Completion": "cancel_ticket"
614
+
},
615
+
{
616
+
"Prompt": "I am having questions regarding the exact time, location, and requirements of the event and would like to talk to customer service.",
617
+
"Completion": "Customer_service"
618
+
}
619
+
]
620
+
,
621
+
prompt="""
622
+
Generate authentic customer support ticket interactions that have a user query and system response.
623
+
For each user query, the system generates a keyword that is used to forward the user to the specific subsystem.
624
+
Requirements for user queries:
625
+
- Use professional, respectful language
626
+
- Follow standard customer service best practices
627
+
Each response should be a single id from the following list:
cancel_ticket means that the customer wants to cancel the ticket.
631
+
customer_service means that customer wants to talk to customer service.
632
+
report_payment_issue means that the customer is facing payment issues and wants to be forwarded to the billing department to resolve the issue.
633
+
634
+
""",
635
+
schema=None
636
+
)
637
+
}
638
+
601
639
602
640
603
641
USE_CASE_CONFIGS_EVALS= {
@@ -916,6 +954,43 @@ class UseCaseMetadataEval(BaseModel):
916
954
917
955
918
956
Give a score rating 1-10 for the given data. If there are more than 9 points to subtract use 1 as the absolute minimum scoring. List all justification as list.
957
+
"""
958
+
),
959
+
UseCase.TICKETING_DATASET: UseCaseMetadataEval(
960
+
name="Ticketing Dataset",
961
+
default_examples=[
962
+
{
963
+
"score": 5,
964
+
"justification": """
965
+
The query is professionally written, respectful, and follows customer service best practices.
966
+
The response 'report_payment_issue' is one of the allowed keywords.
967
+
The matching between the query and response is perfect according to the provided definitions.
968
+
969
+
"""},
970
+
{
971
+
"score": 3,
972
+
"justification": """
973
+
The query is professionally written and respectful.
974
+
The response 'cancel_ticket' is one of the allowed keywords.
975
+
While the response uses a valid keyword, it doesn't match the most appropriate category for the specific query content.
976
+
"""
977
+
},
978
+
979
+
],
980
+
prompt="""
981
+
You are given a user query for a ticketing support system and the system responses which is a keyword that is used to forward the user to the specific subsystem.
982
+
Evaluate whether the queries:
983
+
- Use professional, respectful language
984
+
- Follow standard customer service best practices
985
+
Evaluate whether the responses use only one of the the following keywords: cancel_ticket,customer_service,report_payment_issue
986
+
Evaluate whether the solutions and responses are correctly matched based on the following definitions:
987
+
cancel_ticket means that the customer wants to cancel the ticket.
988
+
customer_service means that customer wants to talk to customer service.
989
+
report_payment_issue means that the customer is facing payment issues and wants to be forwarded to the billing department to resolve the issue.
990
+
Give a score of 1-5 based on the following instructions:
991
+
If the responses don’t match the four keywords give always value 1.
992
+
Rate the quality of the queries and responses based on the instructions give a rating between 1 to 5.
0 commit comments