add evaluation for matching clothes

Author: gonsoomoon-ml

genai/aws-gen-ai-kr/20_applications/05_image_styling_recommendation_with_prompt_engineering/evaluation/README.md

@@ -0,0 +1,57 @@
+# Evaluation for the Image styling recommendation with prompte engineering
+
+## 1. 비즈니스 문제 정의
+- 문제는 크게 아래 두가지로 정의 됩니다. 
+    - (1) 생성형 AI를  통한 어울리는 상품 찾기, 
+        - ![matching_clothes.png](img/matching_clothes.png)
+    - (2) 어울리는 상품을 LLM 이 찾고 "선택 이유" 에 대해서 잘 기술이 되었는지를 체계적으로 검증
+        - ![evaluation_problem.png](img/evaluation_problem.png)
+
+## 2. 솔루션
+notebook 폴더에 아래의 두개의 노트북을 실행하면 솔루션을 얻을 수 있습니다.
+- 01_matching_codi_product.ipynb
+- 02_matching_reason_evaluation.ipynb
+
+## 3.사용 데이터
+- 사용한 이미지는 ["무신사"](https://www.musinsa.com/app/?utm_source=google_shopping&utm_medium=sh&utm_campaign=pmax_ongoing&source=GOSHSAP001&utm_source=google_shopping&utm_medium=sh&utm_campaign=pmax_ongoing&source=GOSHSAP001&gad_source=1&gclid=CjwKCAjw57exBhAsEiwAaIxaZv09yuMwcaiR6VnTCsEtLNv2RGHtxR7uGrDROKAFhzW-rUZst1JCEBoC4I8QAvD_BwE) 의 웹사이트에서 다운로드 한 이미지를 사용합니다.
+
+## 4.실험 환경
+### 4.1 SageMaker Studio Code Editor
+- 노트북은 [SageMaker Studio Code Editor](https://docs.aws.amazon.com/sagemaker/latest/dg/code-editor.html) 및 커널 base (Python 3.10.13) 에서 테스트 되었습니다.
+- 실행 환경에 설치된 Python Package 참고 하세요. --> [requirements.txt](requirements.txt)
+
+### 4.2 기타 환경
+**요구 사항**
+
+* Python 3.7 이상
+* AWS 계정 및 자격 증명
+* AWS CLI 설치 및 구성
+
+**설치**
+
+1. 이 저장소를 클론하세요.
+
+`git clone https://github.com/aws-samples/aws-ai-ml-workshop-kr.git`
+
+가상 환경을 생성하고 활성화합니다.
+
+```bash
+python3 -m venv venv
+source venv/bin/activate
+```
+
+필요한 Python 패키지를 설치합니다.
+
+`pip install -r requirements.txt`
+
+다운로드 받은 깃 리포의 해당 폴더로 이동 합니다.
+`cd genai/aws-gen-ai-kr/20_applications/05_image_styling_recommendation_with_prompt_engineering/evaluation` 
+
+
+## A. 참고 자료
+- [Building with Anthropic’s Claude 3 on Amazon Bedrock and LangChain](https://medium.com/@dminhk/building-with-anthropics-claude-3-on-amazon-bedrock-and-langchain-%EF%B8%8F-2b842f9c0ca8)
+- [Amazon Bedrock 기반 Amorepacific 리뷰 요약 서비스 평가 방법 구현하기](langchain_core.runnables.base.RunnableSequence)
+- [Amazon Bedrock model IDs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html)
+이 저장소에는 Anthropic Claude-3 Sonnet 모델을 AWS Bedrock 런타임에서 사용하는 방법을 보여주는 Python 예제 코드가 포함되어 있습니다.
+- [Anthropic Claude 설명서](https://docs.anthropic.com/claude/docs/intro-to-claude)
+- [AWS Bedrock 런타임 설명서](https://docs.aws.amazon.com/ko_kr/bedrock/latest/userguide/service_code_examples_bedrock-runtime.html)

genai/aws-gen-ai-kr/20_applications/05_image_styling_recommendation_with_prompt_engineering/evaluation/eval_utils/__init__.py

@@ -0,0 +1,163 @@
+import os
+import base64
+import json
+import boto3
+import sys
+import textwrap
+from io import StringIO
+from langchain_core.output_parsers import StrOutputParser
+from langchain_core.prompts import ChatPromptTemplate
+
+
+from langchain_aws import ChatBedrock
+
+class BedrockLangChain:
+
+    def __init__(self, bedrock_runtime):
+        self.bedrock_runtime = bedrock_runtime
+
+    def invoke_rewrite_langchain(self, model_id, model_kwargs, system_prompt, user_prompt, coordination_review, verbose):
+
+        model = ChatBedrock(
+            client=self.bedrock_runtime,
+            model_id= model_id,
+            model_kwargs=model_kwargs,
+        )
+
+
+        messages = [
+            ("system", system_prompt),
+            ("human", user_prompt)
+        ]
+
+        prompt = ChatPromptTemplate.from_messages(messages)
+        if verbose:
+            print("messages: \n", messages)        
+            print("prompt: \n")
+            self.print_ww(prompt)
+
+        chain = prompt | model | StrOutputParser()
+
+        print("## Created Prompt:\n")
+        response = chain.invoke(
+            {
+                "coordination_review": coordination_review
+            }
+        )
+
+        return response
+
+
+
+    def invoke_creating_criteria_langchain(self, model_id, model_kwargs, system_prompt, user_prompt, guide, verbose):
+
+        model = ChatBedrock(
+            client=self.bedrock_runtime,
+            model_id= model_id,
+            model_kwargs=model_kwargs,
+        )
+
+
+        messages = [
+            ("system", system_prompt),
+            ("human", user_prompt)
+        ]
+
+        prompt = ChatPromptTemplate.from_messages(messages)
+        if verbose:
+            print("messages: \n", messages)        
+            print("prompt: \n")
+            self.print_ww(prompt)
+
+        chain = prompt | model | StrOutputParser()
+
+        print("## Created Prompt:\n")
+
+        for chunk in chain.stream(
+            {
+                "guide": guide
+            }
+        ):
+            print(chunk, end="", flush=True)
+
+
+    def invoke_evaluating_fashion_review_langchain(self, model_id, model_kwargs, system_prompt, user_prompt, human_message, AI_message, verbose):
+
+        model = ChatBedrock(
+            client=self.bedrock_runtime,
+            model_id= model_id,
+            model_kwargs=model_kwargs,
+        )
+
+
+        
+        messages = [
+            ("system", system_prompt),
+            ("human", user_prompt)
+        ]
+
+        prompt = ChatPromptTemplate.from_messages(messages)
+        if verbose:
+            print("messages: \n", messages)        
+            print("prompt: \n")
+            self.print_ww(prompt)
+
+        chain = prompt | model | StrOutputParser()
+
+
+        for chunk in chain.stream(
+            {
+                "human_text": human_message,
+                "AI_text": AI_message,                
+            }
+        ):
+            print(chunk, end="", flush=True)
+
+
+    def set_text_langchain_body(self, prompt):
+        text_only_body = {
+                "messages": [
+                    {
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "text",
+                                "text": prompt,
+                            },
+                        ],
+                    }
+                ],
+            }
+        return text_only_body
+    def print_ww(self, *args, width: int = 100, **kwargs):
+        """Like print(), but wraps output to `width` characters (default 100)"""
+        buffer = StringIO()
+        try:
+            _stdout = sys.stdout
+            sys.stdout = buffer
+            print(*args, **kwargs)
+            output = buffer.getvalue()
+        finally:
+            sys.stdout = _stdout
+        for line in output.splitlines():
+            print("\n".join(textwrap.wrap(line, width=width)))
+
+
+
+
+# from langchain.callbacks import StreamlitCallbackHandler
+# model_id="anthropic.claude-3-sonnet-20240229-v1:0", # Claude 3 Sonnet 모델 선택
+# # 텍스트 생성 LLM 가져오기, streaming_callback을 인자로 받아옴
+# def get_llm(boto3_bedrock, model_id):
+#     llm = BedrockChat(
+#     model_id= model_id,
+#     client=boto3_bedrock,
+#     model_kwargs={
+#         "max_tokens": 1024,
+#         "stop_sequences": ["\n\nHuman"],
+#     }
+#     )
+#     return llm
+# llm = get_llm(boto3_bedrock=client, model_id = model_id)
+# response_text = llm.invoke(prompt) #프롬프트에 응답 반환
+# print(response_text.content)

genai/aws-gen-ai-kr/20_applications/05_image_styling_recommendation_with_prompt_engineering/evaluation/eval_utils/langchain_bedrock.py

@@ -0,0 +1,118 @@
+
+class FashionPrompt():
+    def __init__(self):
+        # self.system_prompt = system_prompt
+        pass
+    pass
+
+    def get_rewrite_system_prompt(self):
+        '''
+        주어진 문장을 Re-Write 하는 시스템 프롬프트를 제공 함.
+        '''
+        
+        system_prompt = '''The task is to rewrite a given sentence in a different way while preserving its original meaning.\
+Your role is to take a sentence provided by the user and rephrase it using different words or sentence structures, \
+without altering the core meaning or message conveyed in the original sentence.
+
+Instructions:
+1. Read the sentence carefully and ensure you understand its intended meaning.
+2. Identify the key components of the sentence, such as the subject, verb, object, and any modifiers or additional information.
+3. Think of alternative ways to express the same idea using different vocabulary, sentence structures, or phrasing.
+4. Ensure that your rewritten sentence maintains the same essential meaning as the original, without introducing any new information or altering the original intent.
+5. Pay attention to grammar, punctuation, and overall coherence to ensure your rewritten sentence is well-formed and easy to understand.
+6. If the original sentence contains idioms, metaphors, or cultural references, try to find equivalent expressions or explanations in your rewritten version.
+7. Avoid oversimplifying or overly complicating the sentence; aim for a natural and clear rephrasing that maintains the original tone and complexity.
+
+Remember, the goal is to provide a fresh perspective on the sentence while preserving its core meaning and ensuring clarity and coherence in your rewritten version.
+'''
+
+        return system_prompt
+
+    def get_rewrite_user_prompt(self):
+        '''
+        주어진 문장을 Re-Write 하는 유저 프롬프트를 제공 함.
+        '''
+        
+        user_prompt = '''Given <coordination_review> based on the guide on system prompt         
+Please write in Korean. Output in JSON format following the <output_example> format, excluding <output_example>        
+
+<coordination_review>{coordination_review}</coordination_review>
+<output_example>
+"original_coordination_review" : 
+"rewrite_original_coordination_review" : 
+</output_example>
+'''
+
+        return user_prompt
+
+
+    def get_create_criteria_system_prompt(self):
+        '''
+        주어진 문장을 Re-Write 하는 유저 프롬프트를 제공 함.
+        '''        
+        system_prompt = '''You are a prompt engineering expert.'''
+
+        return system_prompt
+
+    def get_create_criteria_user_prompt(self):
+        
+        user_prompt = '''먼저 당신의 역할과 작업을 XML Tag 없이 기술하세요, \
+이후에 아래의 <guide> 에 맟주어서 프롬프트를 영어로 작성해주세요. 
+<guide>{guide}</guide>'''
+
+        return user_prompt
+
+    def get_fashion_evaluation_system_prompt(self):
+        '''
+        의상 코디에 대한 관련성 여부를 평가 하기 위한 시스템 프롬프트를 제공
+        '''
+
+        
+        system_prompt = '''
+You will be provided with two opinions: one from a fashion expert regarding clothing choices, and \
+another from an AI system offering recommendations on clothing choices. \
+Your task is to evaluate the relevance and coherence between these two opinions \
+by assigning a score from 1 to 5, where 1 indicates low relevance and 5 indicates high relevance.\ 
+You will need to define the criteria for scoring in the <criteria></criteria> section, and \
+outline the steps for evaluating the two opinions in the <steps></steps> section.
+
+<criteria>
+1 - The two opinions are completely unrelated and contradict each other.
+2 - The opinions share some minor similarities, but the overall themes and recommendations are largely different.
+3 - The opinions have moderate overlap in their themes and recommendations, but there are still notable differences.
+4 - The opinions are mostly aligned, with only minor differences in their specific recommendations or perspectives.
+5 - The two opinions are highly coherent, complementary, and provide consistent recommendations or perspectives on clothing choices.
+</criteria>
+
+<steps>
+1. Read and understand the opinion provided by the fashion expert.
+2. Read and understand the opinion provided by the AI system.
+3. Identify the main themes, recommendations, and perspectives presented in each opinion.
+4. Compare the two opinions and assess the degree of alignment or contradiction between them.
+5. Based on the criteria defined above, assign a score from 1 to 5 to reflect the relevance and coherence between the two opinions.
+6. Provide a brief explanation justifying the assigned score.
+</steps>
+'''
+        return system_prompt
+
+    def get_fashion_evaluation_user_prompt(self):
+        '''
+        의상 코디에 대한 관련성 여부를 평가 하기 위한 유저 프롬프트를 제공
+        '''
+        
+        user_prompt = '''
+Given <human_view> and <AI_view>, based on the guide on system prompt         
+Write in the form of <evaluation> in korean with JSON format 
+
+<human_view>{human_text}</human_view>
+<AI_view>{AI_text}</AI_view>
+
+<evaluation> 
+'human_view': 
+'AI_view' :  
+'score': 4,
+'reason': 'AI view is similar to human view'
+</evaluation> 
+'''
+        return user_prompt        
+