Skip to content

Commit 20edbea

Browse files
Shubham-KhichiShubham-Khichi
committed
fix: replace hardcoded localhost URLs with environment variables and other bug fixes
What did we fix? Fixed ECONNREFUSED errors in Docker logs by replacing hardcoded localhost URLs with environment variables What was the actual problem in the codebase? Several files used hardcoded 'http://localhost:24125' URLs for backend API calls In Docker containers, services must be accessed by service name, not localhost This caused connection errors while the application still worked due to error handling What was the solution we implemented? Updated API calls to use process.env.BACKEND_URL with localhost fallback Modified 5 files: storage/route.ts, memory-file/route.ts, all-files/route.ts, crawl-service.ts, MCPServerStatus.tsx Ensures proper service communication in both Docker and local development environments fix(crawler): enhance URL discovery error handling and logging This commit addresses issues with URL discovery not initiating the scraping process by implementing comprehensive error handling and detailed logging throughout the discovery pipeline. Key improvements: - Add connection validation to Crawl4AI service with detailed error reporting - Enhance error handling in discover_pages function to return pages with error status - Implement tracking of consecutive polling errors with graceful fallbacks - Add detailed request/response logging for better debugging - Fix nested try-catch blocks in API routes - Improve frontend error handling with informative user feedback - Add proper TypeScript type annotations for better type safety These changes provide better visibility into the URL discovery process, making it easier to diagnose and fix issues with the crawling pipeline. Error messages are now properly propagated from the backend to the frontend, and users receive more informative feedback when errors occur.
1 parent d9caa76 commit 20edbea

File tree

13 files changed

+437
-58
lines changed

13 files changed

+437
-58
lines changed

‎README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,15 +235,28 @@ DevDocs is more than a tool—it's your documentation companion that:
235235

236236
1. **Open the "Modes" Interface**
237237
- In **Roo Code**, click the **+** to create a new Mode-Specific Prompts.
238+
<br>
239+
238240
2. **Name**
239241
- Give the mode a **Name** (e.g., `Research_MCP`).
242+
<br>
240243
3. **Role Definition Prompt**
244+
<details>
245+
<summary>Prompt</summary>
246+
241247
```
242248
Expertise and Personality: Expertise: Developer documentation retrieval, technical synthesis, and documentation search. Personality: Systematic, detail-oriented, and precise. Provide well-structured answers with clear references to documentation sections.
243249
244250
Behavioral Mandate: Always use the Table Of Contents and Section Access tools when addressing any query regarding the MCP documentation. Maintain clarity, accuracy, and traceability in your responses.
245251
```
252+
</details>
253+
<br>
254+
246255
4. **Mode-Specific Custom Instructions Prompt**
256+
<details>
257+
<summary> Prompt </summary>
258+
259+
247260
```
248261
1. Table Of Contents Tool: Returns a full or filtered list of documentation topics.
249262
2. Section Access Tool: Retrieves the detailed content of specific documentation sections.
@@ -276,6 +289,10 @@ Custom Instruction Loading: Additional custom instructions specific to Research_
276289
277290
Final Output Construction: The final answer should be organized, directly address the query, and include clear pointers (e.g., section names or identifiers) back to the MCP documentation. Ensure minimal redundancy while covering all necessary details.
278291
```
292+
293+
</details>
294+
<br>
295+
279296
## 🤝 Join Our Community
280297

281298
- 🌟 [Star us on GitHub](https://github.com/cyberagi/devdocs)

‎app/api/all-files/route.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,8 @@ export async function GET(request: Request) {
9090
// Get in-memory files from the backend
9191
let memoryFiles = []
9292
try {
93-
const memoryResponse = await fetch('http://localhost:24125/api/memory-files')
93+
const backendUrl = process.env.NEXT_PUBLIC_BACKEND_URL || process.env.BACKEND_URL || 'http://localhost:24125'
94+
const memoryResponse = await fetch(`${backendUrl}/api/memory-files`)
9495
if (memoryResponse.ok) {
9596
const memoryData = await memoryResponse.json()
9697
if (memoryData.success && Array.isArray(memoryData.files)) {

‎app/api/discover/route.ts‎

Lines changed: 64 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
1-
import { NextResponse } from 'next/server'
2-
import { discoverSubdomains } from '@/lib/crawl-service'
1+
import { NextResponse } from 'next/server'
32

43
export async function POST(request: Request) {
54
try {
@@ -16,21 +15,77 @@ export async function POST(request: Request) {
1615
const validatedDepth = Math.min(5, Math.max(1, parseInt(String(depth)) || 3))
1716

1817
console.log('Making discover request for URL:', url, 'with depth:', validatedDepth)
19-
const pages = await discoverSubdomains({ url, depth: validatedDepth })
20-
console.log('Received pages from backend:', pages)
18+
19+
// Make a direct request to the backend API instead of using the discoverSubdomains function
20+
const backendUrl = process.env.NEXT_PUBLIC_BACKEND_URL || process.env.BACKEND_URL || 'http://localhost:24125'
21+
console.log('Using backend URL:', backendUrl)
22+
23+
console.log('Sending request to backend API:', `${backendUrl}/api/discover`)
24+
const response = await fetch(`${backendUrl}/api/discover`, {
25+
method: 'POST',
26+
headers: {
27+
'Content-Type': 'application/json',
28+
},
29+
body: JSON.stringify({ url, depth: validatedDepth }),
30+
})
31+
32+
console.log('Response status from backend:', response.status)
33+
34+
if (!response.ok) {
35+
const errorData = await response.json()
36+
console.error('Error response from backend:', errorData)
37+
return NextResponse.json(
38+
{ error: errorData.error || 'Failed to discover pages' },
39+
{ status: response.status }
40+
)
41+
}
42+
43+
const data = await response.json()
44+
console.log('Received response from backend:', data)
45+
console.log('Discovered pages count:', data.pages?.length || 0)
46+
if (data.pages?.length > 0) {
47+
console.log('First discovered page:', data.pages[0])
48+
} else {
49+
console.warn('No pages were discovered')
50+
}
2151

2252
// Even if we get an empty array, we should still return it with a 200 status
23-
return NextResponse.json({
24-
pages,
25-
message: pages.length === 0 ? 'No pages discovered' : `Found ${pages.length} pages`
53+
return NextResponse.json({
54+
pages: data.pages || [],
55+
message: data.message || (data.pages?.length === 0 ? 'No pages discovered' : `Found ${data.pages?.length} pages`)
2656
})
2757

2858
} catch (error) {
2959
console.error('Error in discover route:', error)
60+
61+
// Log more detailed information about the error
62+
if (error instanceof Error) {
63+
console.error('Error name:', error.name)
64+
console.error('Error message:', error.message)
65+
console.error('Error stack:', error.stack)
66+
67+
// Check for network-related errors
68+
if (error.message.includes('fetch') || error.message.includes('network') || error.message.includes('ECONNREFUSED')) {
69+
console.error('Network error detected - possible connection issue to backend service')
70+
}
71+
72+
// Check for timeout errors
73+
if (error.message.includes('timeout')) {
74+
console.error('Timeout error detected - backend service might be taking too long to respond')
75+
}
76+
}
77+
78+
// Check if it's a TypeError, which might indicate issues with the response format
79+
if (error instanceof TypeError) {
80+
console.error('TypeError detected - possible issue with response format or undefined values')
81+
}
82+
3083
return NextResponse.json(
31-
{
84+
{
3285
error: error instanceof Error ? error.message : 'Failed to discover pages',
33-
details: error instanceof Error ? error.stack : undefined
86+
details: error instanceof Error ? error.stack : undefined,
87+
errorType: error instanceof Error ? error.name : 'Unknown',
88+
pages: []
3489
},
3590
{ status: 500 }
3691
)

‎app/api/memory-file/route.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@ export async function GET(request: Request) {
1313
}
1414

1515
// Fetch the in-memory file from the backend
16-
const response = await fetch(`http://localhost:24125/api/memory-files/${id}`)
16+
const backendUrl = process.env.NEXT_PUBLIC_BACKEND_URL || process.env.BACKEND_URL || 'http://localhost:24125'
17+
const response = await fetch(`${backendUrl}/api/memory-files/${id}`)
1718

1819
if (!response.ok) {
1920
const errorData = await response.json()

‎app/api/storage/route.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,8 @@ export async function GET(request: Request) {
165165
// Get in-memory files from the backend
166166
let memoryFiles = []
167167
try {
168-
const memoryResponse = await fetch('http://localhost:24125/api/memory-files')
168+
const backendUrl = process.env.NEXT_PUBLIC_BACKEND_URL || process.env.BACKEND_URL || 'http://localhost:24125'
169+
const memoryResponse = await fetch(`${backendUrl}/api/memory-files`)
169170
if (memoryResponse.ok) {
170171
const memoryData = await memoryResponse.json()
171172
if (memoryData.success && Array.isArray(memoryData.files)) {

‎app/page.tsx‎

Lines changed: 120 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -43,26 +43,108 @@ export default function Home() {
4343

4444
try {
4545
console.log('Discovering pages for:', submittedUrl, 'with depth:', depth)
46-
const pages = await discoverSubdomains({ url: submittedUrl, depth })
47-
console.log('Discovered pages:', pages)
46+
47+
// Make a direct request to the Next.js API route instead of the backend directly
48+
// This provides better error handling and logging
49+
console.log('Making request to Next.js API route: /api/discover')
50+
51+
const response = await fetch(`/api/discover`, {
52+
method: 'POST',
53+
headers: {
54+
'Content-Type': 'application/json',
55+
},
56+
body: JSON.stringify({ url: submittedUrl, depth }),
57+
})
58+
59+
console.log('Response status:', response.status)
60+
61+
// Get the response data
62+
const data = await response.json()
63+
console.log('Response data:', data)
64+
65+
if (!response.ok) {
66+
// Extract detailed error information if available
67+
const errorMessage = data.error || 'Failed to discover pages'
68+
const errorDetails = data.details || ''
69+
const errorType = data.errorType || 'Unknown'
70+
71+
console.error('Error response:', {
72+
message: errorMessage,
73+
details: errorDetails,
74+
type: errorType
75+
})
76+
77+
// Show a more detailed error toast
78+
toast({
79+
title: `Error: ${errorType}`,
80+
description: errorMessage,
81+
variant: "destructive"
82+
})
83+
84+
throw new Error(errorMessage)
85+
}
86+
87+
const pages = data.pages || []
88+
console.log('Discovered pages count:', pages.length)
89+
90+
if (pages.length > 0) {
91+
console.log('First discovered page:', pages[0])
92+
93+
// Check if any pages have error status
94+
const errorPages = pages.filter((page: DiscoveredPage) => page.status === 'error')
95+
if (errorPages.length > 0) {
96+
console.warn(`${errorPages.length} pages have error status`)
97+
98+
// Show a warning toast if some pages have errors
99+
toast({
100+
title: "Warning",
101+
description: `${errorPages.length} pages encountered errors during discovery`,
102+
variant: "default"
103+
})
104+
}
105+
} else {
106+
console.warn('No pages were discovered')
107+
108+
// Show a warning toast if no pages were discovered
109+
toast({
110+
title: "No Pages Found",
111+
description: "The crawler couldn't find any pages to process. This might be due to access restrictions or an invalid URL structure.",
112+
variant: "default"
113+
})
114+
}
48115

49116
setDiscoveredPages(pages)
50117
setStats(prev => ({
51118
...prev,
52-
subdomainsParsed: pages.length
119+
subdomainsParsed: pages.length,
120+
errorsEncountered: pages.filter((page: DiscoveredPage) => page.status === 'error').length
53121
}))
54122

55-
toast({
56-
title: "Pages Discovered",
57-
description: `Found ${pages.length} related pages at depth ${depth}`
58-
})
59-
} catch (error) {
123+
// Only show success toast if pages were actually discovered
124+
if (pages.length > 0) {
125+
toast({
126+
title: "Pages Discovered",
127+
description: `Found ${pages.length} related pages at depth ${depth}`
128+
})
129+
}
130+
} catch (error: unknown) {
60131
console.error('Error discovering pages:', error)
61-
toast({
62-
title: "Error",
63-
description: error instanceof Error ? error.message : "Failed to discover pages",
64-
variant: "destructive"
65-
})
132+
133+
// If we haven't already shown an error toast from the response data
134+
if (error instanceof Error && !error.message.includes('Failed to discover pages')) {
135+
toast({
136+
title: "Error",
137+
description: error.message || "Failed to discover pages. Check the console for more details.",
138+
variant: "destructive"
139+
})
140+
} else {
141+
// Fallback for non-Error objects
142+
toast({
143+
title: "Error",
144+
description: "Failed to discover pages. Check the console for more details.",
145+
variant: "destructive"
146+
})
147+
}
66148
} finally {
67149
setIsProcessing(false)
68150
}
@@ -86,7 +168,31 @@ export default function Home() {
86168
}))
87169
)
88170

89-
const result = await crawlPages(selectedPages)
171+
// Make a direct request to the backend API instead of using the crawlPages function
172+
const backendUrl = process.env.NEXT_PUBLIC_BACKEND_URL || process.env.BACKEND_URL || 'http://localhost:24125'
173+
console.log('Making direct request to backend:', `${backendUrl}/api/crawl`)
174+
175+
const response = await fetch(`${backendUrl}/api/crawl`, {
176+
method: 'POST',
177+
headers: {
178+
'Content-Type': 'application/json',
179+
},
180+
body: JSON.stringify({ pages: selectedPages }),
181+
})
182+
183+
if (!response.ok) {
184+
const errorData = await response.json()
185+
throw new Error(errorData.error || 'Failed to crawl pages')
186+
}
187+
188+
const data = await response.json()
189+
console.log('Crawl response data:', data)
190+
191+
const result = {
192+
markdown: data.markdown || '',
193+
links: data.links || { internal: [], external: [] },
194+
error: data.error
195+
}
90196
console.log('Crawl result:', result)
91197

92198
if (result.error) {

0 commit comments

Comments
 (0)