Use Presidio for masking (#40)

* Use Presidio for masking
* Improve PII to handle encoded content
* Reject large encoded content as DOS
* Handle hex errors
* Fix structured output masking path

Author: steven10a

docs/ref/checks/competitors.md

@@ -30,11 +30,9 @@ Returns a `GuardrailResult` with the following `info` dictionary:
 {
     "guardrail_name": "Competitor Detection",
     "competitors_found": ["competitor1"],
-    "checked_competitors": ["competitor1", "rival-company.com"],
-    "checked_text": "Original input text"
+    "checked_competitors": ["competitor1", "rival-company.com"]
 }
 ```
 
 - **`competitors_found`**: List of competitors detected in the text
 - **`checked_competitors`**: List of competitors that were configured for detection
-- **`checked_text`**: Original input text

docs/ref/checks/custom_prompt_check.md

@@ -35,12 +35,10 @@ Returns a `GuardrailResult` with the following `info` dictionary:
     "guardrail_name": "Custom Prompt Check",
     "flagged": true,
     "confidence": 0.85,
-    "threshold": 0.7,
-    "checked_text": "Original input text"
+    "threshold": 0.7
 }
 ```
 
 - **`flagged`**: Whether the custom validation criteria were met
 - **`confidence`**: Confidence score (0.0 to 1.0) for the validation
 - **`threshold`**: The confidence threshold that was configured
-- **`checked_text`**: Original input text

docs/ref/checks/hallucination_detection.md

@@ -113,8 +113,7 @@ Returns a `GuardrailResult` with the following `info` dictionary:
     "hallucination_type": "factual_error",
     "hallucinated_statements": ["Our premium plan costs $299/month"],
     "verified_statements": ["We offer customer support"],
-    "threshold": 0.7,
-    "checked_text": "Our premium plan costs $299/month and we offer customer support"
+    "threshold": 0.7
 }
 ```
 
@@ -125,7 +124,6 @@ Returns a `GuardrailResult` with the following `info` dictionary:
 - **`hallucinated_statements`**: Specific statements that are contradicted or unsupported
 - **`verified_statements`**: Statements that are supported by your documents
 - **`threshold`**: The confidence threshold that was configured
-- **`checked_text`**: Original input text
 
 Tip: `hallucination_type` is typically one of `factual_error`, `unsupported_claim`, or `none`.
 

docs/ref/checks/jailbreak.md

@@ -56,15 +56,13 @@ Returns a `GuardrailResult` with the following `info` dictionary:
     "guardrail_name": "Jailbreak",
     "flagged": true,
     "confidence": 0.85,
-    "threshold": 0.7,
-    "checked_text": "Original input text"
+    "threshold": 0.7
 }
 ```
 
 - **`flagged`**: Whether a jailbreak attempt was detected
 - **`confidence`**: Confidence score (0.0 to 1.0) for the detection
 - **`threshold`**: The confidence threshold that was configured
-- **`checked_text`**: Original input text
 
 ## Related checks
 

docs/ref/checks/keywords.md

@@ -25,11 +25,9 @@ Returns a `GuardrailResult` with the following `info` dictionary:
 {
     "guardrail_name": "Keyword Filter",
     "matched": ["confidential", "secret"],
-    "checked": ["confidential", "secret", "internal only"],
-    "checked_text": "This is confidential information that should be kept secret"
+    "checked": ["confidential", "secret", "internal only"]
 }
 ```
 
 - **`matched`**: List of keywords found in the text
 - **`checked`**: List of keywords that were configured for detection
-- **`checked_text`**: Original input text

docs/ref/checks/moderation.md

@@ -57,12 +57,10 @@ Returns a `GuardrailResult` with the following `info` dictionary:
         "violence": 0.12,
         "self-harm": 0.08,
         "sexual": 0.03
-    },
-    "checked_text": "Original input text"
+    }
 }
 ```
 
 - **`flagged`**: Whether any category violation was detected
 - **`categories`**: Boolean flags for each category indicating violations
 - **`category_scores`**: Confidence scores (0.0 to 1.0) for each category
-- **`checked_text`**: Original input text

docs/ref/checks/nsfw.md

@@ -44,15 +44,13 @@ Returns a `GuardrailResult` with the following `info` dictionary:
     "guardrail_name": "NSFW Text",
     "flagged": true,
     "confidence": 0.85,
-    "threshold": 0.7,
-    "checked_text": "Original input text"
+    "threshold": 0.7
 }
 ```
 
 - **`flagged`**: Whether NSFW content was detected
 - **`confidence`**: Confidence score (0.0 to 1.0) for the detection
 - **`threshold`**: The confidence threshold that was configured
-- **`checked_text`**: Original input text
 
 ### Examples
 

docs/ref/checks/off_topic_prompts.md

@@ -35,12 +35,10 @@ Returns a `GuardrailResult` with the following `info` dictionary:
     "guardrail_name": "Off Topic Prompts",
     "flagged": false,
     "confidence": 0.85,
-    "threshold": 0.7,
-    "checked_text": "Original input text"
+    "threshold": 0.7
 }
 ```
 
 - **`flagged`**: Whether the content aligns with your business scope
 - **`confidence`**: Confidence score (0.0 to 1.0) for the prompt injection detection assessment
 - **`threshold`**: The confidence threshold that was configured
-- **`checked_text`**: Original input text

docs/ref/checks/pii.md

@@ -2,22 +2,33 @@
 
 Detects personally identifiable information (PII) such as SSNs, phone numbers, credit card numbers, and email addresses using Microsoft's [Presidio library](https://microsoft.github.io/presidio/). Will automatically mask detected PII or block content based on configuration.
 
+**Advanced Security Features:**
+
+- **Unicode normalization**: Prevents bypasses using fullwidth characters (＠) or zero-width spaces
+- **Encoded PII detection**: Optionally detects PII hidden in Base64, URL-encoded, or hex strings
+- **URL context awareness**: Detects emails in query parameters (e.g., `GET /api?user=john@example.com`)
+- **Custom recognizers**: Includes CVV/CVC codes and BIC/SWIFT codes beyond Presidio defaults
+
 ## Configuration
 
 ```json
 {
     "name": "Contains PII",
     "config": {
-        "entities": ["EMAIL_ADDRESS", "US_SSN", "CREDIT_CARD", "PHONE_NUMBER"],
-        "block": false
+        "entities": ["EMAIL_ADDRESS", "US_SSN", "CREDIT_CARD", "PHONE_NUMBER", "CVV", "BIC_SWIFT"],
+        "block": false,
+        "detect_encoded_pii": false
     }
 }
 ```
 
 ### Parameters
 
-- **`entities`** (required): List of PII entity types to detect. See the full list of [supported entities](https://microsoft.github.io/presidio/supported_entities/).
+- **`entities`** (required): List of PII entity types to detect. Includes:
+    - Standard Presidio entities: See the full list of [supported entities](https://microsoft.github.io/presidio/supported_entities/)
+    - Custom entities: `CVV` (credit card security codes), `BIC_SWIFT` (bank identification codes)
 - **`block`** (optional): Whether to block content or just mask PII (default: `false`)
+- **`detect_encoded_pii`** (optional): If `true`, detects PII in Base64/URL-encoded/hex strings (default: `false`)
 
 ## Implementation Notes
 
@@ -41,6 +52,8 @@ Detects personally identifiable information (PII) such as SSNs, phone numbers, c
 
 Returns a `GuardrailResult` with the following `info` dictionary:
 
+### Basic Example (Plain PII)
+
 ```json
 {
     "guardrail_name": "Contains PII",
@@ -55,8 +68,34 @@ Returns a `GuardrailResult` with the following `info` dictionary:
 }
 ```
 
-- **`detected_entities`**: Detected entities and their values
+### With Encoded PII Detection Enabled
+
+When `detect_encoded_pii: true`, the guardrail also detects and masks encoded PII:
+
+```json
+{
+    "guardrail_name": "Contains PII",
+    "detected_entities": {
+        "EMAIL_ADDRESS": [
+            "user@email.com",
+            "am9obkBleGFtcGxlLmNvbQ==",
+            "%6a%6f%65%40domain.com",
+            "6a6f686e406578616d706c652e636f6d"
+        ]
+    },
+    "entity_types_checked": ["EMAIL_ADDRESS"],
+    "checked_text": "Contact <EMAIL_ADDRESS> or <EMAIL_ADDRESS_ENCODED> or <EMAIL_ADDRESS_ENCODED>",
+    "block_mode": false,
+    "pii_detected": true
+}
+```
+
+Note: Encoded PII is masked with `<ENTITY_TYPE_ENCODED>` to distinguish it from plain text PII.
+
+### Field Descriptions
+
+- **`detected_entities`**: Detected entities and their values (includes both plain and encoded forms when `detect_encoded_pii` is enabled)
 - **`entity_types_checked`**: List of entity types that were configured for detection
-- **`checked_text`**: Text with PII masked (if PII was found) or original text (if no PII was found)
+- **`checked_text`**: Text with PII masked. Plain PII uses `<ENTITY_TYPE>`, encoded PII uses `<ENTITY_TYPE_ENCODED>`
 - **`block_mode`**: Whether the check was configured to block or mask
-- **`pii_detected`**: Boolean indicating if any PII was found
+- **`pii_detected`**: Boolean indicating if any PII was found (plain or encoded)

docs/ref/checks/prompt_injection_detection.md

@@ -73,8 +73,7 @@ Returns a `GuardrailResult` with the following `info` dictionary:
             "name": "get_weather",
             "arguments": "{'location': 'Tokyo'}"
         }
-    ],
-    "checked_text": "[{'role': 'user', 'content': 'What is the weather in Tokyo?'}]"
+    ]
 }
 ```
 
@@ -84,7 +83,6 @@ Returns a `GuardrailResult` with the following `info` dictionary:
 - **`threshold`**: The confidence threshold that was configured
 - **`user_goal`**: The tracked user intent from conversation
 - **`action`**: The list of function calls or tool outputs analyzed for alignment
-- **`checked_text`**: Serialized conversation history inspected during analysis
 
 ## Benchmark Results