Modified Crawling and graceful error handling with streamlined UI. (#48)

Patch Fixes: 

- Fixed MCP Docker Build Failure: Resolved the build error for the mcp
service by removing the invalid readme reference in
fast-markdown-mcp/pyproject.toml.
- Refactored File Handling (Removed In-Memory Storage):
- Investigated the complex in-memory file handling mechanism and its
inconsistencies.
- Removed the in-memory storage logic from backend/app/crawler.py.
- Removed the associated API endpoints (/api/memory-files,
/api/memory-files/{file_id}) from backend/app/main.py.
- Added a new backend API endpoint (/api/storage/file-content) to read
files directly from the storage/markdown directory.
- Deleted the old frontend API proxy route
(app/api/memory-file/route.ts).
- Created a new frontend API proxy route
(app/api/storage/file-content/route.ts).
- Updated frontend components (StoredFiles.tsx, DiscoveredFiles.tsx) to
use the new API route for downloading file content.
- Documentation: Created markdown plans for the MCP build fix and the
in-memory feature removal.
- This simplifies the architecture by relying solely on disk-based
consolidated files in storage/markdown. Please remember to test the file
download functionality after restarting the services.

feat: Enhance crawl workflow, UI, and fix backend issues
This commit addresses several issues and implements enhancements across
the crawling workflow:

Fixes:
- Resolved 400 Bad Request error caused by incorrect query parameter
(`file_path`) in the file content API route.
- Fixed backend `NameError` (`set_task_context`) in crawler.py that
prevented result file saving.
- Corrected 500 Internal Server Error caused by Docker networking issue
(localhost vs. service name) in the file content API route proxy.
- Ensured 'Data Extracted' statistic is correctly saved in the backend
status and displayed in the UI.

UI Enhancements:
- Made "Consolidated Files" section persistent, rendering as soon as a
job ID is available.
- Relocated "Crawl Selected" button inline with status details.
- Updated "Crawl Selected" button to show dynamic count and disable
appropriately.
- Renamed "Job Status" section title to "Discovered Pages".
- Renamed "Processing Summary" section title to "Statistics".
- Removed the unused "Extracted Content" display section.

Backend Enhancements:
- Implemented file appending logic in crawler.py for consolidated `.md`
and `.json` files. Subsequent crawls for the same job now append data
and update timestamps instead of overwriting.


feat(frontend): Update Consolidated Files component for polling and
downloads

- Implements polling every 10 seconds in ConsolidatedFiles.tsx to
automatically refresh the list of files from the /api/storage endpoint,
ensuring newly added files appear in the UI.
- Modifies the MD and JSON icon links to point to the
/api/storage/download endpoint and adds the 'download' attribute,
triggering file downloads instead of opening content in the browser.

Author: Shubham-Khichi

.roo/rules-boomerang/rules.md

@@ -0,0 +1,3 @@
+Special Rules for Critiquing plans and strategies:
+
+Once you have created a subtask markdown file or in memory, you are to get a second opinion from the mode called "Expert Opinion" This mode is designed to only accept your subtask plans and strategies before you present it to the user to approve. You are to do this before you deligate to either Code or Debug modes and after you have created a subtask. Consider this mode as your personal brainstorm. Argue with it from ground truth about the codebase, both you and the Expert Opinion mode as complete knowledge of the codebase. I want you to counter the points and come to a common understanding for the most accurate and right path moving forward and then present those findings to the user. 

.roomodes

@@ -0,0 +1,17 @@
+{
+  "customModes": [
+    {
+      "slug": "boomerang",
+      "name": "boomerang ",
+      "roleDefinition": "You are Roo, a strategic workflow orchestrator who coordinates complex tasks by delegating them to appropriate specialized modes. You have a comprehensive understanding of each mode's capabilities and limitations, allowing you to effectively break down complex problems into discrete tasks that can be solved by different specialists.\n\nProcedure to follow is to ask the coder for an implementation plan without writing any code yet, forward the implementation plan to the Expert Opinion mode for review, and based the feedback from Expert Opinion mode give the coder the go-ahead to create a updated task list and then ask for user to approve the task list which has the feedbac of Expert Opinion mode and previous plans with pros and cons for each. Your prompts for those roles will have to ensure they know the procedure too, so they’re all singing from the same hymn-sheet.",
+      "customInstructions": "Your role is to coordinate complex workflows by delegating tasks to specialized modes. As an orchestrator, you should:\n\n1. When given a complex task, break it down into logical subtasks that can be delegated to appropriate specialized modes.\n\n2. For each subtask, use the `new_task` tool to delegate. Choose the most appropriate mode for the subtask's specific goal and provide comprehensive instructions in the `message` parameter. These instructions must include:\n    *   All necessary context from the parent task or previous subtasks required to complete the work.\n    *   A clearly defined scope, specifying exactly what the subtask should accomplish.\n    *   An explicit statement that the subtask should *only* perform the work outlined in these instructions and not deviate.\n    *   An instruction for the subtask to signal completion by using the `attempt_completion` tool, providing a concise yet thorough summary of the outcome in the `result` parameter, keeping in mind that this summary will be the source of truth used to keep track of what was completed on this project.\n    *   A statement that these specific instructions supersede any conflicting general instructions the subtask's mode might have.\n    * Once you have the plan created by the coder  forward the implementation plan to the Expert Opinion mode for review, and based on the result ask for improvements or give the coder the go-ahead. Your prompts for those roles will have to ensure they know the procedure too, so they’re all singing from the same hymn-sheet.\n\n3. Track and manage the progress of all subtasks in a markdown file in the codebase. If its a bug then start the heading with BUG: if its a feature then write FEATURE:. When a subtask is completed, analyze its results from the user and determine the next steps and then go back to complete the markdown file subtask. \n\n4. Help the user understand how the different subtasks fit together in the overall workflow. Provide clear reasoning about why you're delegating specific tasks to specific modes.\n\n5. When all subtasks are completed, synthesize the results and provide a comprehensive overview of what was accomplished.\n\n6. Always Ask clarifying questions when necessary to better understand how to break down complex tasks effectively in as much detail as possible. \n\n7. Suggest improvements to the workflow based on the results of completed subtasks.\n\nUse subtasks to maintain clarity. If a request significantly shifts focus or requires a different expertise (mode), consider creating a subtask rather than overloading the current one.",
+      "groups": [
+        "read",
+        "edit",
+        "command",
+        "mcp"
+      ],
+      "source": "project"
+    }
+  ]
+}

README.md

@@ -1,8 +1,8 @@
 # DevDocs by CyberAGI 🚀
 
 <div align="center">
-  <img src="https://github.com/user-attachments/assets/6d4cc4df-fe5d-4483-9218-3d621f572e49" alt="DevDocs Interface" width="800">
-  <img src="https://github.com/user-attachments/assets/00350dc6-2ff3-40cf-b0b3-8b3e387d983d" alt="DevDocs Interface" width="800">
+  <img src="assets/image.png" alt="DevDocs Interface" width="800">
+
 
   <p align="center">
     <strong>Turn Weeks of Documentation Research into Hours of Productive Development</strong>
@@ -108,18 +108,43 @@ git clone https://github.com/cyberagiinc/DevDocs.git
 # Navigate to the project directory
 cd DevDocs
 
+# Configure environment variables
+# Copy the template file to .env
+cp .env.template .env
+
+# Ensure NEXT_PUBLIC_BACKEND_URL in .env is set correctly (e.g., http://localhost:24125)
+# This allows the frontend (running in your browser) to communicate with the backend service.
+
+
 # Start all services using Docker
 ./docker-start.sh
 ```
 
-For Windows users:
+For Windows users: Experimental Only (Not Tested Yet)
 ```cmd
 # Clone the repository
 git clone https://github.com/cyberagiinc/DevDocs.git
 
 # Navigate to the project directory
+
 cd DevDocs
 
+# Configure environment variables
+# Copy the template file to .env
+
+copy .env.template .env
+
+# Ensure NEXT_PUBLIC_BACKEND_URL in .env is set correctly (e.g., http://localhost:24125)
+
+# This allows the frontend (running in your browser) to communicate with the backend service.
+
+# Prerequisites: Install WSL 2 and Docker Desktop
+# Docker Desktop for Windows requires WSL 2. Please ensure you have WSL 2 installed and running first.
+# 1. Install WSL 2: Follow the official Microsoft guide: https://learn.microsoft.com/en-us/windows/wsl/install
+# 2. Install Docker Desktop for Windows: Download and install from the official Docker website. Docker Desktop includes Docker Compose.
+
+
+
 # Start all services using Docker
 docker-start.bat
 ```

app/api/all-files/route.ts

@@ -90,8 +90,7 @@ export async function GET(request: Request) {
     // Get in-memory files from the backend
     let memoryFiles = []
     try {
-      const backendUrl = process.env.NEXT_PUBLIC_BACKEND_URL || process.env.BACKEND_URL || 'http://localhost:24125'
-      const memoryResponse = await fetch(`${backendUrl}/api/memory-files`)
+      const memoryResponse = await fetch('http://backend:24125/api/memory-files')
       if (memoryResponse.ok) {
         const memoryData = await memoryResponse.json()
         if (memoryData.success && Array.isArray(memoryData.files)) {

app/api/memory-file/route.ts

@@ -0,0 +1,74 @@
+import { NextResponse } from 'next/server'
+import { type NextRequest } from 'next/server'
+
+// Determine backend URL (consider containerized vs. local development)
+// In Docker, use the service name 'backend'. Locally, use localhost.
+// Force using the service name for Docker context, as env var might not be reliable here.
+const backendHost = 'http://backend:24125';
+
+export async function GET(request: NextRequest) {
+  try {
+    const searchParams = request.nextUrl.searchParams;
+    const filePath = searchParams.get('file_path'); // Expecting file_path relative to storage/markdown
+
+    if (!filePath) {
+      return NextResponse.json(
+        { success: false, error: 'Missing file path parameter' },
+        { status: 400 }
+      );
+    }
+
+    // Basic validation/sanitization (more robust checks might be needed depending on usage)
+    if (filePath.includes('..')) {
+       return NextResponse.json(
+        { success: false, error: 'Invalid file path' },
+        { status: 400 }
+      );
+    }
+
+    // Construct the backend URL
+    // The backend endpoint expects 'file_path' query parameter
+    const backendUrl = new URL(`${backendHost}/api/storage/file-content`);
+    backendUrl.searchParams.append('file_path', filePath);
+
+    console.log(`Fetching from backend: ${backendUrl.toString()}`); // Log the backend URL being called
+
+    // Fetch the file content from the backend
+    const response = await fetch(backendUrl.toString());
+
+    if (!response.ok) {
+      let errorData;
+      try {
+        // Try to parse JSON error from backend first
+        errorData = await response.json();
+      } catch (parseError) {
+        // If backend didn't send JSON, use text
+        errorData = { error: await response.text() };
+      }
+      console.error(`Backend fetch failed (${response.status}):`, errorData.error);
+      return NextResponse.json(
+        { success: false, error: errorData.error || `Failed to fetch file content (Status: ${response.status})` },
+        { status: response.status }
+      );
+    }
+
+    // Get content as text (backend returns PlainTextResponse)
+    const content = await response.text();
+
+    // Return content directly (assuming frontend handles rendering)
+    // Set appropriate content type if needed, e.g., 'text/markdown'
+    return new NextResponse(content, {
+      status: 200,
+      headers: {
+        'Content-Type': filePath.endsWith('.json') ? 'application/json' : 'text/plain; charset=utf-8', // Adjust content type based on file extension
+      },
+    });
+
+  } catch (error) {
+    console.error('Error fetching storage file content:', error);
+    return NextResponse.json(
+      { success: false, error: error instanceof Error ? error.message : 'Failed to fetch file content' },
+      { status: 500 }
+    );
+  }
+}

app/api/storage/file-content/route.ts

@@ -41,8 +41,23 @@ export async function GET(request: Request) {
       const mdFiles = files.filter(f => f.endsWith('.md'))
       const jsonFiles = files.filter(f => f.endsWith('.json'))
 
+      // Define interface for disk file details
+      interface DiskFileDetail {
+        name: string;
+        jsonPath: string;
+        markdownPath: string;
+        timestamp: Date;
+        size: number;
+        wordCount: number;
+        charCount: number;
+        isConsolidated: boolean;
+        pagesCount: number;
+        rootUrl: string;
+        isInMemory: boolean;
+      }
+
       // Get disk files
-      const diskFileDetails = await Promise.all(
+      const diskFileDetails: DiskFileDetail[] = await Promise.all(
         mdFiles.map(async (filename) => {
           const mdPath = path.join(STORAGE_DIR, filename)
           const jsonPath = path.join(STORAGE_DIR, filename.replace('.md', '.json'))
@@ -162,42 +177,13 @@ export async function GET(request: Request) {
         metadata?: any;
       }
 
-      // Get in-memory files from the backend
-      let memoryFiles = []
-      try {
-        const backendUrl = process.env.NEXT_PUBLIC_BACKEND_URL || process.env.BACKEND_URL || 'http://localhost:24125'
-        const memoryResponse = await fetch(`${backendUrl}/api/memory-files`)
-        if (memoryResponse.ok) {
-          const memoryData = await memoryResponse.json()
-          if (memoryData.success && Array.isArray(memoryData.files)) {
-            // Convert in-memory files to the same format as disk files
-            memoryFiles = memoryData.files
-              .filter((file: MemoryFile) => !file.isJson) // Only include markdown files
-              .map((file: MemoryFile) => ({
-                name: file.name,
-                jsonPath: file.path.replace('.md', '.json'),
-                markdownPath: file.path,
-                timestamp: new Date(file.timestamp),
-                size: file.size,
-                wordCount: file.wordCount,
-                charCount: file.charCount,
-                isConsolidated: false,
-                pagesCount: 1,
-                rootUrl: '',
-                isInMemory: true
-              }))
-          }
-        }
-      } catch (e) {
-        console.error('Error fetching in-memory files:', e)
-      }
-      
-      // Combine disk and memory files
-      const allFiles = [...diskFileDetails, ...memoryFiles]
-      
+      // Removed fetching and combining of in-memory files as that feature was removed.
+      // We now only work with files read from disk.
+      const allFiles = diskFileDetails // Keep variable name for minimal diff, even though it's just disk files now
+
       // Filter out individual files (non-consolidated files)
       // Only show consolidated files in the Stored Files section
-      const consolidatedFiles = allFiles.filter(file => file.isConsolidated)
+      const consolidatedFiles = allFiles.filter((file: DiskFileDetail) => file.isConsolidated)
 
       // Additional filter to exclude files with UUID-like names
       // UUID pattern: 8-4-4-4-12 hex digits (e.g., 095104d8-8e90-48f0-8670-9e45c914f115)