Feat/twilio devtools (#4)

This PR introduces comprehensive Twilio DevTools integration for the kernel browser project, adding secure credential management and enhanced development capabilities. The main purpose is to integrate Twilio's Network Traversal Service for TURN credentials and add Chrome DevTools frontend support for better debugging.

- Adds Twilio TURN credential management with dynamic generation and fallback handling
- Integrates enhanced Chrome DevTools frontend with dedicated service configuration  
- Implements secure credential storage using Google Secret Manager with automated deployment workflows

Author: olesho

.env.example

@@ -0,0 +1,17 @@
+# Twilio Network Traversal Service Credentials
+# Get these from your Twilio Console:
+# 1. Go to https://console.twilio.com/
+# 2. Navigate to Account > API Keys & Tokens
+# 3. Create a new API Key
+# 4. Use the SID as TWILIO_ACCOUNT_SID
+# 5. Use the Secret as TWILIO_AUTH_TOKEN
+TWILIO_ACCOUNT_SID=SK...your_api_key_sid_here
+TWILIO_AUTH_TOKEN=your_api_key_secret_here
+
+# Google Cloud Configuration
+# If not provided, will use current gcloud config
+PROJECT_ID=your-gcp-project-id
+# REGION=us-central1
+
+# Optional: Service Configuration
+# SERVICE_NAME=kernel-browser

.gitignore

@@ -0,0 +1,54 @@
+# Environment variables
+.env
+.env.local
+*.env
+!.env.example
+
+# Node modules
+node_modules/
+
+# Build outputs
+dist/
+build/
+out/
+
+# Logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# OS files
+.DS_Store
+Thumbs.db
+
+# IDE files
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# Temporary files
+tmp/
+temp/
+*.tmp
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+.Python
+venv/
+env/
+
+# Google Cloud
+.gcloudignore
+gcs-key.json
+service-account-key.json
+
+# Docker
+.dockerignore
+
+# Backup files
+*.bak
+*.backup

DEPLOYMENT.md

@@ -0,0 +1,166 @@
+# Kernel Browser - Cloud Run Deployment Guide
+
+This guide explains how to deploy the Kernel Browser to Google Cloud Run with secure Twilio credential management.
+
+## Prerequisites
+
+- Google Cloud SDK (`gcloud`) installed
+- Docker installed
+- Git installed
+- A Google Cloud Project with billing enabled
+- Twilio account with API credentials (for WebRTC TURN servers)
+
+## Quick Start
+
+### 1. Clone the repository
+```bash
+git clone <repository-url>
+cd browser-web-agent
+git submodule update --init --recursive
+```
+
+### 2. Set up Twilio credentials
+```bash
+# Copy the example environment file
+cp .env.example .env
+
+# Edit .env and add your Twilio credentials
+# Get these from https://console.twilio.com/ > Account > API Keys & Tokens
+```
+
+Your `.env` file should contain:
+```
+TWILIO_ACCOUNT_SID=SK...your_api_key_sid_here
+TWILIO_AUTH_TOKEN=your_api_key_secret_here
+```
+
+### 3. Deploy to Cloud Run
+```bash
+./deploy.sh
+```
+
+The script will:
+- Load credentials from `.env`
+- Create/update secrets in Google Secret Manager
+- Build and deploy the container to Cloud Run
+- Configure all necessary permissions
+
+## Deployment Options
+
+### Using Cloud Build (recommended)
+```bash
+./deploy.sh
+```
+
+### Using local Docker build
+```bash
+./deploy.sh --local
+```
+
+### Specify project and region
+```bash
+./deploy.sh --project YOUR_PROJECT_ID --region us-central1
+```
+
+## How It Works
+
+### Credential Management
+
+1. **Local Development**: Credentials are stored in `.env` file (gitignored)
+2. **Secret Manager**: Deploy script automatically creates/updates secrets in Google Secret Manager
+3. **Cloud Run**: Service uses `secretKeyRef` to securely access credentials at runtime
+4. **Dynamic TURN**: Container fetches fresh TURN credentials from Twilio on startup
+
+### Security Features
+
+- Credentials never appear in code or logs
+- Secrets are encrypted at rest and in transit
+- Service account has minimal required permissions
+- Automatic credential rotation support
+
+### Files Overview
+
+- `.env.example` - Template for environment variables
+- `.env` - Your local credentials (gitignored)
+- `deploy.sh` - Main deployment script with Secret Manager integration
+- `service-secrets.yaml` - Cloud Run config with secret references
+- `service.yaml` - Fallback config (for deployments without secrets)
+- `cloudbuild.yaml` - Cloud Build configuration
+- `twilio/` - Twilio credential management scripts
+
+## Updating Credentials
+
+To update Twilio credentials:
+
+1. Update `.env` with new credentials
+2. Run `./deploy.sh` again
+3. Script will update secrets and redeploy
+
+## Manual Secret Management
+
+If you need to manage secrets manually:
+
+```bash
+# Create secrets
+echo -n "YOUR_SID" | gcloud secrets create twilio-account-sid --data-file=-
+echo -n "YOUR_TOKEN" | gcloud secrets create twilio-auth-token --data-file=-
+
+# Update secrets
+echo -n "NEW_SID" | gcloud secrets versions add twilio-account-sid --data-file=-
+echo -n "NEW_TOKEN" | gcloud secrets versions add twilio-auth-token --data-file=-
+
+# Grant access to service account
+gcloud secrets add-iam-policy-binding twilio-account-sid \
+  --member="serviceAccount:kernel-browser-sa@PROJECT_ID.iam.gserviceaccount.com" \
+  --role="roles/secretmanager.secretAccessor"
+```
+
+## Service Endpoints
+
+After deployment, you'll have access to:
+
+- **Main Interface**: `https://SERVICE_URL/`
+- **WebRTC Client**: `https://SERVICE_URL/`
+- **Chrome DevTools**: `https://SERVICE_URL/devtools/`
+- **DevTools WebSocket**: `wss://SERVICE_URL/cdp/ws`
+- **Recording API**: `https://SERVICE_URL/api`
+- **Health Check**: `https://SERVICE_URL/health`
+
+## Troubleshooting
+
+### Deployment fails
+- Check that all prerequisites are installed
+- Ensure billing is enabled on your GCP project
+- Verify you have sufficient quota in your region
+
+### WebRTC not working
+- Ensure Twilio credentials are correct
+- Check Cloud Run logs: `gcloud run services logs read kernel-browser --region=us-central1`
+- Verify TURN servers are accessible from your network
+
+### Secrets not found
+- Run `gcloud secrets list` to verify secrets exist
+- Check service account permissions
+- Ensure Secret Manager API is enabled
+
+## Architecture
+
+```
+┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
+│   Client    │────▶│   Cloud Run      │────▶│ Secret Manager  │
+│  (Browser)  │     │   (Container)    │     │  (Credentials)  │
+└─────────────┘     └──────────────────┘     └─────────────────┘
+                            │
+                            ▼
+                    ┌──────────────────┐
+                    │   Twilio API     │
+                    │  (TURN Servers)  │
+                    └──────────────────┘
+```
+
+## Support
+
+For issues or questions:
+- Check logs: `gcloud run services logs read kernel-browser --region=us-central1`
+- Review service status: `gcloud run services describe kernel-browser --region=us-central1`
+- File an issue on GitHub

Dockerfile.cloudrun

@@ -1,3 +1,57 @@
+# DevTools Frontend build stage using browser-operator-core
+FROM --platform=linux/amd64 ubuntu:22.04 AS devtools-builder
+
+# Cache bust argument to force rebuilds
+ARG CACHE_BUST
+
+# Install required packages for DevTools frontend build
+RUN apt-get update && apt-get install -y \
+    curl \
+    git \
+    python3 \
+    python3-pip \
+    python-is-python3 \
+    wget \
+    unzip \
+    sudo \
+    ca-certificates \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Node.js 18.x
+RUN curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
+    apt-get install -y nodejs && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /workspace
+
+# Clone depot_tools
+RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+ENV PATH="/workspace/depot_tools:${PATH}"
+ENV DEPOT_TOOLS_UPDATE=0
+
+# Follow README instructions exactly - fetching code
+RUN mkdir devtools
+WORKDIR /workspace/devtools
+RUN fetch devtools-frontend
+
+# Build steps
+WORKDIR /workspace/devtools/devtools-frontend
+
+RUN gclient sync
+RUN /workspace/depot_tools/ensure_bootstrap
+
+# Build standard DevTools first
+RUN npm run build
+
+# Add Browser Operator fork and switch to it
+RUN git remote add upstream https://github.com/BrowserOperator/browser-operator-core.git
+RUN git fetch upstream
+RUN git checkout upstream/main
+
+# Build Browser Operator version
+RUN npm run build
+
 # Multi-stage build using kernel-images as base
 FROM docker.io/golang:1.25.0 AS server-builder
 WORKDIR /workspace/server
@@ -90,6 +144,12 @@ RUN apt-get update && \
     nginx \
     # PPA req
     software-properties-common && \
+    # Disable nginx auto-start to prevent conflicts with custom config
+    systemctl disable nginx || true && \
+    systemctl mask nginx || true && \
+    # Remove default nginx config to prevent conflicts
+    rm -f /etc/nginx/sites-enabled/default && \
+    rm -f /etc/nginx/nginx.conf && \
     # Userland apps
     sudo add-apt-repository ppa:mozillateam/ppa && \
     sudo apt-get install -y --no-install-recommends \
@@ -186,19 +246,40 @@ COPY kernel-images/images/chromium-headful/supervisor/services/ /etc/supervisor/
 # Copy the kernel-images API binary
 COPY --from=server-builder /out/kernel-images-api /usr/local/bin/kernel-images-api
 
-# Cloud Run specific: nginx configuration for port proxying
-COPY nginx.conf /etc/nginx/nginx.conf
+# ============================================================================
+# DevTools Integration
+# ============================================================================
+
+# Copy DevTools static files from builder
+COPY --from=devtools-builder /workspace/devtools/devtools-frontend/out/Default/gen/front_end /usr/share/nginx/devtools
+
+# Set permissions for DevTools files
+RUN chown -R kernel:kernel /usr/share/nginx/devtools
+
+# Cloud Run specific: wrapper scripts (nginx config is inline)
+# DO NOT copy nginx.conf to avoid auto-start conflicts
 COPY cloudrun-wrapper.sh /cloudrun-wrapper.sh
-RUN chmod +x /cloudrun-wrapper.sh
+COPY twilio/twilio-credential-updater.sh /twilio-credential-updater.sh
+RUN chmod +x /cloudrun-wrapper.sh /twilio-credential-updater.sh
+
+# Add essential services for neko WebRTC and Chromium
+COPY supervisor/services-cloudrun/dbus.conf /etc/supervisor/conf.d/services-cloudrun/dbus.conf
+COPY supervisor/services-cloudrun/xorg.conf /etc/supervisor/conf.d/services-cloudrun/xorg.conf
+COPY supervisor/services-cloudrun/neko.conf /etc/supervisor/conf.d/services-cloudrun/neko.conf
+COPY supervisor/services-cloudrun/chromium.conf /etc/supervisor/conf.d/services-cloudrun/chromium.conf
+COPY supervisor/services-cloudrun/devtools-frontend.conf /etc/supervisor/conf.d/services-cloudrun/devtools-frontend.conf
 
 # Create nginx temp directories for non-root execution
 RUN mkdir -p /tmp/nginx_client_temp /tmp/nginx_proxy_temp /tmp/nginx_fastcgi_temp \
-    /tmp/nginx_uwsgi_temp /tmp/nginx_scgi_temp && \
+    /tmp/nginx_uwsgi_temp /tmp/nginx_scgi_temp \
+    /tmp/nginx_devtools_client_temp /tmp/nginx_devtools_proxy_temp /tmp/nginx_devtools_fastcgi_temp \
+    /tmp/nginx_devtools_uwsgi_temp /tmp/nginx_devtools_scgi_temp && \
     chown -R kernel:kernel /tmp/nginx_*
 
 # Create supervisor log directories
 RUN mkdir -p /var/log/supervisord/chromium /var/log/supervisord/neko /var/log/supervisord/xorg \
-    /var/log/supervisord/dbus /var/log/supervisord/kernel-images-api /var/log/supervisord/mutter && \
+    /var/log/supervisord/dbus /var/log/supervisord/kernel-images-api /var/log/supervisord/mutter \
+    /var/log/supervisord/nginx /var/log/supervisord/devtools-frontend && \
     chown -R kernel:kernel /var/log/supervisord
 
 # Create health check endpoint