Add llama.cpp local inference support for Mac/local users #222
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add support for running DeepResearch locally using llama.cpp with Metal (Apple Silicon) or CUDA acceleration. Zero API costs, full privacy. New files: - inference/interactive_llamacpp.py: ReAct agent CLI for llama.cpp - scripts/start_llama_server.sh: Server startup script with optimized settings - requirements-local.txt: Minimal dependencies for local inference Features: - Free web search via DuckDuckGo (no API key required) - Optional Jina Reader for better page content extraction - Loop detection to prevent infinite tool call cycles - 32K context window for long research sessions - Exponential backoff retry for rate limits - URL validation before visiting pages Works with bartowski's GGUF quantizations (~18GB for Q4_K_M).
- Add Exa, Tavily, Serper, DuckDuckGo providers with automatic fallback - Remove emojis for cleaner professional output - Update documentation with search provider information - Show available search providers on startup
- Add Tavily as 2nd provider (Exa → Tavily → Serper → DuckDuckGo) - Use Bearer token auth for Tavily (per API docs) - Add quota error handling (Exa 402, Tavily 432/433) - Add sanitize_query() for input validation - Switch Serper from http.client to requests - Handle ConnectionError, JSONDecodeError, DuckDuckGoSearchException - Use Exa type: auto for better search quality
- Change default context size from 32K to 16K (saves ~8GB RAM) - Disable --mlock by default (prevents locking 18GB model in wired memory) - Add --mlock flag to opt-in if needed for performance - Add --low-memory flag for constrained systems (8K context) - Display mlock status in configuration output This prevents the system from running out of memory when running llama-server alongside other apps like Firefox.
- Reduce MAX_ROUNDS from 30 to 10 for faster, more practical research queries - Add --alias flag to llama-server for cleaner model naming (fixes Open WebUI compatibility) - Better UX when using with web UIs like Open WebUI that expect short model names
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add support for running DeepResearch 100% locally using llama.cpp with Metal (Apple Silicon) or CUDA acceleration. Zero API costs, full privacy.
Why This?
The main inference path requires vLLM with 8x A100 GPUs. This PR adds an alternative for:
New Files
inference/interactive_llamacpp.pyscripts/start_llama_server.shrequirements-local.txtrequests,duckduckgo-search,python-dotenvFeatures
Requirements
-DLLAMA_METAL=ON) or CUDA supportQuick Start
Testing
Tested on Apple M1 Max with 32GB RAM:
Related
This is a cleaner alternative to PR #220 (MLX support), which had issues with chat template handling. llama.cpp is more mature and widely used.