Add MCP well-known URI auth discovery per MCP Spec (#2527)

* Add MCP well-known URI auth discovery

Implements RFC 9728 Protected Resource Metadata discovery via
well-known URIs when WWW-Authenticate header is not present.
This completes ToolHive's implementation of the MCP specification
requirement that clients MUST support both discovery mechanisms.

Changes:
- Add tryWellKnownDiscovery() to discover auth via well-known URIs
- Add buildWellKnownURI() to construct RFC 9728 compliant URIs
- Add checkWellKnownURIExists() to validate URI accessibility
- Modify DetectAuthenticationFromServer() for well-known fallback
- Add comprehensive unit tests for all discovery paths
- Add regression tests for empty scope handling contracts

The implementation tries endpoint-specific URIs first, then root-level
URIs per MCP spec priority order. This enables authentication with
MCP-compliant servers that use well-known URIs per RFC 9728 Section 3
but don't send WWW-Authenticate headers.

Test coverage: 75.5% for pkg/auth/discovery

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Jon Christiansen <467023+theJC@users.noreply.github.com>

* Update per MR feedback regarding content-type returned from wellknown endpoint, also dont consider wellknown as existing if we cannot access due to unauthorized/401

* fix golangci-lint identified issues

* Add more tests for coverage

* Update comment to make clearer

* PR PR feedback, defensively control how much response draining we are willing to do

* Update test to be more effecient per PR feedback

---------

Signed-off-by: Jon Christiansen <467023+theJC@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>

Author: theJC

docs/remote-mcp-authentication.md

@@ -16,12 +16,29 @@ ToolHive is **highly compliant** with the MCP authorization specification, imple
 - Extracts `realm` and `resource_metadata` parameters as per RFC 9728
 - Handles error and error_description parameters
 
-#### 2. Protected Resource Metadata (RFC 9728)
-- **Location**: [`pkg/auth/discovery/discovery.go:531-593`](../pkg/auth/discovery/discovery.go#L531)
-- Fetches metadata from `resource_metadata` URL in WWW-Authenticate header
+#### 2. Protected Resource Metadata Discovery (RFC 9728 & MCP Specification)
+
+ToolHive implements BOTH discovery mechanisms required by the MCP specification:
+
+**Method 1: WWW-Authenticate Header (Primary)**
+- **Location**: [`pkg/auth/discovery/discovery.go:148-156`](../pkg/auth/discovery/discovery.go#L148)
+- Extracts `resource_metadata` parameter from `Bearer` scheme in WWW-Authenticate header
+- Takes precedence when present (most efficient path)
+
+**Method 2: Well-Known URI Fallback (MCP Specification Requirement)**
+- **Location**: [`pkg/auth/discovery/discovery.go:176-254`](../pkg/auth/discovery/discovery.go#L176)
+- **Specification**: [MCP Protected Resource Metadata Discovery Requirements](https://modelcontextprotocol.io/specification/draft/basic/authorization#protected-resource-metadata-discovery-requirements)
+- Triggers when no WWW-Authenticate header present
+- Tries endpoint-specific URI: `/.well-known/oauth-protected-resource/{path}`
+- Falls back to root-level URI: `/.well-known/oauth-protected-resource`
+- Uses HTTP GET per RFC 9728 requirement
+
+**Metadata Processing (Common to Both Methods)**
+- **Location**: [`pkg/auth/discovery/discovery.go:575-637`](../pkg/auth/discovery/discovery.go#L575)
 - Validates HTTPS requirement (with localhost exception for development)
 - Verifies required `resource` field presence
 - Extracts and processes `authorization_servers` array
+- Enables automatic discovery for servers that only implement well-known URIs
 
 #### 3. Authorization Server Discovery (RFC 8414)
 - **Location**: [`pkg/auth/discovery/discovery.go:595-621`](../pkg/auth/discovery/discovery.go#L595)
@@ -48,16 +65,29 @@ When ToolHive connects to a remote MCP server ([`pkg/runner/remote_auth.go:27-87
 
 1. Makes test request to the remote server (GET, then optionally POST)
 2. Checks for 401 Unauthorized response with WWW-Authenticate header
-3. Parses authentication requirements from the header
+3. **If WWW-Authenticate header found:** Parses authentication requirements from the header
+4. **If no WWW-Authenticate header:** Falls back to RFC 9728 well-known URI discovery:
+   - Tries `{baseURL}/.well-known/oauth-protected-resource/{path}` (endpoint-specific)
+   - Falls back to `{baseURL}/.well-known/oauth-protected-resource` (root-level)
 
 ### Discovery Priority Chain
 ToolHive follows this priority order for discovering the OAuth issuer ([`pkg/runner/remote_auth.go:95-145`](../pkg/runner/remote_auth.go#L95)):
 
-1. **Configured Issuer**: Uses `--remote-auth-issuer` flag if provided
-2. **Realm-Derived**: Derives from `realm` parameter in WWW-Authenticate header (RFC 8414)
-3. **Resource Metadata**: Fetches from `resource_metadata` URL (RFC 9728)
-4. **Well-Known Discovery**: Probes server's well-known endpoints to discover actual issuer (handles issuer mismatch)
-5. **URL-Derived**: Falls back to deriving from the remote URL (last resort)
+**Phase 1: WWW-Authenticate Header Detection**
+1. **Configured Issuer**: Uses `--remote-auth-issuer` flag if provided (highest priority)
+2. **WWW-Authenticate Header**: Checks for `Bearer` scheme with:
+   - **Realm-Derived**: Derives from `realm` parameter (RFC 8414)
+   - **Resource Metadata**: Fetches from `resource_metadata` URL (RFC 9728)
+
+**Phase 2: Well-Known URI Fallback (MCP Specification Requirement)**
+When no WWW-Authenticate header is present, tries RFC 9728 well-known URIs:
+3. **Endpoint-Specific Well-Known URI**: `{baseURL}/.well-known/oauth-protected-resource/{path}`
+4. **Root-Level Well-Known URI**: `{baseURL}/.well-known/oauth-protected-resource`
+5. **Authorization Server Discovery**: Validates each server in metadata via OIDC discovery
+6. **Issuer Mismatch Handling**: Accepts authoritative issuer from well-known endpoints per RFC 8414
+
+**Phase 3: Fallback Discovery**
+7. **URL-Derived**: Falls back to deriving from the remote URL (last resort)
 
 ### Authentication Branches
 
@@ -66,32 +96,40 @@ graph TD
     A[Remote MCP Server Request] --> B{401 Response?}
     B -->|No| C[No Authentication Required]
     B -->|Yes| D{WWW-Authenticate Header?}
-    D -->|No| E[No Authentication Required]
     D -->|Yes| F{Parse Header}
-    
+
+    %% NEW: Well-known URI fallback when no WWW-Authenticate
+    D -->|No| WK1[Try Well-Known URI Discovery]
+    WK1 --> WK2{Try Endpoint-Specific URI}
+    WK2 -->|Found| WK4[Extract Auth Info]
+    WK2 -->|404| WK3{Try Root-Level URI}
+    WK3 -->|Found| WK4
+    WK3 -->|404| E[No Authentication Required]
+    WK4 --> K[Fetch Resource Metadata]
+
     F --> G{Has Realm URL?}
     G -->|Yes| H[Derive Issuer from Realm]
     H --> I[OIDC Discovery]
-    
+
     F --> J{Has resource_metadata?}
-    J -->|Yes| K[Fetch Resource Metadata]
+    J -->|Yes| K
     K --> L[Validate Auth Servers]
     L --> M[Use First Valid Server]
-    
+
     F --> S{No Realm/Metadata?}
     S -->|Yes| T[Probe Well-Known Endpoints]
     T --> U{Found Valid Issuer?}
     U -->|Yes| V[Use Discovered Issuer]
     U -->|No| W[Derive from URL]
-    
+
     I --> N{Client Credentials?}
     M --> N
     V --> N
     W --> N
     N -->|No| O[Dynamic Registration]
     N -->|Yes| P[OAuth Flow]
     O --> P
-    
+
     P --> Q[Get Access Token]
     Q --> R[Authenticated Request]
 ```
@@ -120,16 +158,66 @@ When `resource_metadata` URL is provided:
    - Uses first valid server found
 4. **Handle Issuer Mismatch**: Supports cases where metadata URL differs from actual issuer
 
-## Well-Known Endpoint Discovery
+## Well-Known URI Discovery (RFC 9728 & MCP Specification)
+
+ToolHive implements the MCP specification's **Protected Resource Metadata Discovery Requirements**, which mandates trying well-known URIs when no WWW-Authenticate header is present.
+
+### Discovery Process
+
+**When to Trigger:**
+- Server returns 401 Unauthorized
+- No WWW-Authenticate header in response
+- No manual `--remote-auth-issuer` configured
 
-When no realm URL or resource metadata is provided ([`pkg/runner/remote_auth.go:175-211`](../pkg/runner/remote_auth.go#L175)):
+**Discovery Sequence** ([`pkg/auth/discovery/discovery.go:222-254`](../pkg/auth/discovery/discovery.go#L222)):
 
-1. **Derive Base URL**: Creates a base URL from the server URL
-2. **Probe Well-Known Endpoints**: Attempts to fetch OAuth metadata without requiring issuer match
-3. **Accept Authoritative Issuer**: Uses the issuer from the well-known response as authoritative per RFC 8414
-4. **Log Mismatch**: Records when discovered issuer differs from server URL for debugging
+Per MCP spec priority, ToolHive tries well-known URIs in this order:
+
+1. **Endpoint-Specific URI**: `{baseURL}/.well-known/oauth-protected-resource/{original-path}`
+   - Example: For `https://mcp.example.com/api/v1/mcp`
+   - Tries: `https://mcp.example.com/.well-known/oauth-protected-resource/api/v1/mcp`
+
+2. **Root-Level URI**: `{baseURL}/.well-known/oauth-protected-resource`
+   - Example: For `https://mcp.example.com/api/v1/mcp`
+   - Falls back to: `https://mcp.example.com/.well-known/oauth-protected-resource`
+
+**HTTP Method:**
+- Uses `GET` requests per RFC 9728 requirement
+- Sets `Accept: application/json` header
+- Validates `Content-Type: application/json` header in response
+- Returns on first successful response (200 OK only - metadata must be publicly accessible)
+
+**Response Processing:**
+- Extracts `authorization_servers` array from metadata
+- Validates each authorization server via OIDC discovery
+- Uses first valid server found
+- Accepts authoritative issuer from well-known response per RFC 8414
+
+**Example: Server with Well-Known URI Only**
+
+Some MCP servers implement RFC 9728 well-known URI but don't send WWW-Authenticate headers:
+
+```bash
+# Request to MCP endpoint
+GET https://mcp.example.com/api/v1/mcp
+→ 401 Unauthorized (no WWW-Authenticate header)
+
+# Well-known URI fallback (root-level)
+GET https://mcp.example.com/.well-known/oauth-protected-resource
+→ 200 OK
+
+# Response
+{
+  "resource": "https://mcp.example.com",
+  "authorization_servers": ["https://auth.example.com"],
+  "bearer_methods_supported": ["header"]
+}
+
+# Result
+ToolHive automatically discovers and authenticates without manual configuration
+```
 
-This approach handles cases where the OAuth provider's issuer differs from the server's public URL, such as when using CDN or worker deployments.
+This approach handles cases where servers implement RFC 9728 well-known URI discovery but don't send WWW-Authenticate headers, making authentication completely automatic.
 
 ## Dynamic Client Registration Flow
 
@@ -325,7 +413,8 @@ The `oauth_config` section supports:
 
 | Specification | Status | Implementation |
 |--------------|--------|----------------|
-| RFC 9728 (Protected Resource Metadata) | ✅ Compliant | Full implementation with validation |
+| RFC 9728 (Protected Resource Metadata) | ✅ Fully Compliant | WWW-Authenticate + well-known URI fallback |
+| MCP Well-Known URI Fallback | ✅ Compliant | Tries endpoint-specific and root-level URIs per spec |
 | RFC 8414 (Authorization Server Metadata) | ✅ Compliant | Accepts authoritative issuer from well-known endpoints |
 | RFC 7591 (Dynamic Client Registration) | ✅ Compliant | Automatic registration when needed |
 | OAuth 2.1 PKCE | ✅ Compliant | Enabled by default |

pkg/auth/discovery/discovery.go

@@ -34,6 +34,7 @@ const (
 	DefaultAuthDetectTimeout = 10 * time.Second
 	MaxRetryAttempts         = 3
 	RetryBaseDelay           = 2 * time.Second
+	MaxResponseBodyDrain     = 1 * 1024 * 1024 // 1 MB - limit response body draining to prevent resource exhaustion
 )
 
 // AuthInfo contains authentication information extracted from WWW-Authenticate header
@@ -112,6 +113,21 @@ func DetectAuthenticationFromServer(ctx context.Context, targetURI string, confi
 		}
 	}
 
+	// NEW: Well-known URI fallback per MCP specification
+	// When no WWW-Authenticate header found, try well-known URIs
+	logger.Debugf("No WWW-Authenticate header found, attempting well-known URI discovery")
+
+	wellKnownAuthInfo, err := tryWellKnownDiscovery(detectCtx, client, targetURI)
+	if err != nil {
+		logger.Debugf("Well-known URI discovery failed: %v", err)
+		return nil, nil // Not an error, just no auth detected
+	}
+
+	if wellKnownAuthInfo != nil {
+		logger.Infof("Discovered authentication via well-known URI")
+		return wellKnownAuthInfo, nil
+	}
+
 	return nil, nil // No authentication required
 }
 
@@ -156,6 +172,104 @@ func detectAuthWithRequest(
 	return nil, nil
 }
 
+// buildWellKnownURI constructs a well-known URI for OAuth Protected Resource metadata
+// per RFC 9728 Section 3.1 and MCP specification
+func buildWellKnownURI(parsedURL *url.URL, endpointSpecific bool) string {
+	baseURL := url.URL{
+		Scheme: parsedURL.Scheme,
+		Host:   parsedURL.Host,
+	}
+
+	if endpointSpecific && parsedURL.Path != "" && parsedURL.Path != "/" {
+		// Endpoint-specific: /.well-known/oauth-protected-resource/<original-path>
+		// Remove leading slash from original path to avoid double slashes
+		cleanPath := strings.TrimPrefix(parsedURL.Path, "/")
+		baseURL.Path = path.Join(auth.WellKnownOAuthResourcePath, cleanPath)
+	} else {
+		// Root-level: /.well-known/oauth-protected-resource
+		baseURL.Path = auth.WellKnownOAuthResourcePath
+	}
+
+	return baseURL.String()
+}
+
+// checkWellKnownURIExists returns true if a well-known URI is accessible and returns application/json
+// Per RFC 9728, protected resource metadata MUST be queried using HTTP GET and MUST return application/json
+func checkWellKnownURIExists(ctx context.Context, client *http.Client, uri string) bool {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, uri, nil)
+	if err != nil {
+		logger.Debugf("Failed to create GET request for %s: %v", uri, err)
+		return false
+	}
+
+	req.Header.Set("Accept", "application/json")
+
+	resp, err := client.Do(req)
+	if err != nil {
+		logger.Debugf("Failed to check %s: %v", uri, err)
+		return false
+	}
+	defer func() {
+		// Drain and close response body to enable connection reuse
+		// Limit draining to MaxResponseBodyDrain to prevent resource exhaustion from large responses
+		_, _ = io.CopyN(io.Discard, resp.Body, MaxResponseBodyDrain)
+		_ = resp.Body.Close()
+	}()
+
+	// RFC 9728 requires 200 OK status code - metadata endpoints must be publicly accessible
+	if resp.StatusCode != http.StatusOK {
+		return false
+	}
+
+	// RFC 9728 requires Content-Type to be application/json
+	contentType := strings.ToLower(resp.Header.Get("Content-Type"))
+	if !strings.Contains(contentType, "application/json") {
+		logger.Debugf("Well-known URI %s returned unexpected content type: %s", uri, contentType)
+		return false
+	}
+
+	return true
+}
+
+// tryWellKnownDiscovery attempts to discover authentication requirements via well-known URIs
+// per MCP specification Section: Protected Resource Metadata Discovery Requirements.
+// Tries endpoint-specific path first, then root-level path.
+func tryWellKnownDiscovery(ctx context.Context, client *http.Client, targetURI string) (*AuthInfo, error) {
+	parsedURL, err := url.Parse(targetURI)
+	if err != nil {
+		return nil, fmt.Errorf("invalid target URI: %w", err)
+	}
+
+	// Build well-known URIs to try (in priority order per MCP spec)
+	wellKnownURIs := []string{
+		// 1. Endpoint-specific: /.well-known/oauth-protected-resource/<path>
+		buildWellKnownURI(parsedURL, true),
+		// 2. Root-level: /.well-known/oauth-protected-resource
+		buildWellKnownURI(parsedURL, false),
+	}
+
+	// Try each well-known URI in order
+	for _, wellKnownURI := range wellKnownURIs {
+		logger.Debugf("Trying well-known URI: %s", wellKnownURI)
+
+		// Check if the URI exists before attempting to fetch
+		if !checkWellKnownURIExists(ctx, client, wellKnownURI) {
+			logger.Debugf("Well-known URI not found: %s", wellKnownURI)
+			continue
+		}
+
+		// URI exists - return AuthInfo with ResourceMetadata set
+		// Downstream handler will use FetchResourceMetadata to get the actual metadata
+		logger.Infof("Found well-known URI: %s", wellKnownURI)
+		return &AuthInfo{
+			Type:             "OAuth",
+			ResourceMetadata: wellKnownURI,
+		}, nil
+	}
+
+	return nil, nil // No well-known metadata found
+}
+
 // ParseWWWAuthenticate parses the WWW-Authenticate header to extract authentication information
 // Supports multiple authentication schemes and complex header formats
 func ParseWWWAuthenticate(header string) (*AuthInfo, error) {