Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #315
Changes Proposed
Added database as a new choice for the --save argument
Location: main.py lines 138-139
Created src/torbot/modules/database.py
Implements SearchResultsDatabase class for SQLite management
No external database server required (uses built-in sqlite3)
Added saveDatabase() method in src/torbot/modules/linktree.py (lines 159-195)
Extracts all discovered links and metadata for persistent storage
Created src/torbot/modules/db_query.py for result retrieval
Created scripts/query_database.py CLI for database operations
Explanation of Changes
Database Engine & Architecture
SQLite (file-based, no server)<project_root>/torbot_search_results.db
Auto-initialized on first use
Database Schema
searches Table (Search Metadata)
links Table (Individual Link Records)
Relationship: One search has many links (1:N relationship with CASCADE delete)
Metadata Captured Per Search
Root-Level Metadata:
✅ Root URL being crawled
✅ Exact timestamp of search (ISO 8601)
✅ Crawl depth configuration
✅ Total link count
Per-Link Metadata:
✅ Full URL
✅ Page title
✅ HTTP status code (connectivity indicator)
✅ Content classification (marketplace, forum, etc.)
✅ Classification accuracy/confidence
✅ Email addresses extracted
✅ Phone numbers extracted
Core Features:
Usage
Basic Save:
Benefits: