Claude/update readme tech stack 011o u4 r4f fe nd4 n aqvhgn9 nb

[datagram1]

Purpose

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[mergify]

Documentation preview: https://vllm--28738.org.readthedocs.build/en/28738/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @datagram1.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gemini-code-assist]

Code Review

This pull request introduces significant new functionality for CPU KV cache offloading, including installation scripts for multiple platforms and a web-based management UI. The core logic changes for offloading and scheduler deadlocks appear to be solid bug fixes. However, the implementation of the installation scripts and the management UI introduces several critical and high-severity issues. Specifically, the Windows installer insecurely downloads an executable, the Linux installer uses a brittle patching method, and the management UI relies on fragile log parsing for metrics and has unhandled sudo requirements. These issues should be addressed to ensure the security, correctness, and maintainability of the new features.

[gemini-code-assist]

The script downloads nssm.exe from an external URL and executes it without verifying its integrity. This creates a significant security vulnerability. If the nssm.cc domain or the zip file is compromised, an attacker could execute arbitrary code on the user's machine. It is critical to verify the checksum (e.g., SHA256) of the downloaded file against a known-good value.

$nssmUrl = "https://nssm.cc/release/nssm-2.24.zip"
$expectedHash = "c5d85c53373a09b4a03c456be83153355f547d33923346596150533566106612" # SHA256 for nssm-2.24.zip
$nssmZip = "$env:TEMP\nssm.zip"
Invoke-WebRequest -Uri $nssmUrl -OutFile $nssmZip
$actualHash = (Get-FileHash -Path $nssmZip -Algorithm SHA256).Hash.ToLower()
if ($actualHash -ne $expectedHash) {
    Write-Error-Message "NSSM download hash mismatch! Expected: $expectedHash, Got: $actualHash. Aborting installation."
    Remove-Item $nssmZip
    exit 1
}
Write-Success "NSSM download verified."
Expand-Archive -Path $nssmZip -DestinationPath $env:TEMP -Force
Copy-Item "$env:TEMP\nssm-2.24\win64\nssm.exe" $nssmPath
Remove-Item $nssmZip
Remove-Item "$env:TEMP\nssm-2.24" -Recurse

[gemini-code-assist]

The manager relies on parsing the log file at the hardcoded path /var/log/vllm.log to extract real-time monitoring statistics. This is a critical anti-pattern for several reasons:

1. Brittleness: Parsing metrics from human-readable log lines with regular expressions is extremely fragile. Any change in the log format, even minor ones, will break the metrics collection.
2. Tight Coupling: The manager is tightly coupled to the systemd service configuration, assuming the log file will always be at /var/log/vllm.log. This makes the setup inflexible.
3. Inefficiency: Reading and parsing a log file repeatedly is inefficient for metrics collection.

A much more robust and standard approach is to expose metrics via a dedicated endpoint, for example, using a library like Prometheus. The engine should expose its internal state through a structured API, not through unstructured logs.

[gemini-code-assist]

The apply_cpu_offload_fix function patches the installed vllm library by directly copying a file into the site-packages directory. This approach is very brittle and not portable. It relies on the internal directory structure of Python's virtual environments, which can vary between Python versions, operating systems, and pip versions. A more robust solution would be to install the forked version of vllm directly from the source code, which would include all necessary patches.

[gemini-code-assist]

This script uses monkey-patching to inject the KVTransferConfig into the vLLM engine. This is a fragile approach that depends on the internal implementation details of vllm.engine.arg_utils.EngineArgs.create_engine_config. Any changes to this function in future vLLM versions could break this script silently. It would be more robust to pass the configuration through a supported API, if one exists, or to modify the entrypoint to accept it directly. If this is the only way, it should be heavily commented with warnings about its brittleness.

[gemini-code-assist]

The run_systemctl_command function executes sudo systemctl ... to manage the vllm.service. This assumes that the user running the vllm_manager.py script has passwordless sudo privileges for systemctl. If not, the command will hang waiting for a password, or fail. This is a significant usability and security issue. The requirement for passwordless sudo should be clearly documented in INSTALL_README.md and README.md, along with instructions on how to configure it securely (e.g., via sudoers file to only allow specific commands).

[gemini-code-assist]

The original condition if not preempted_reqs: could cause a deadlock where, after preemption, no waiting requests would be scheduled, leaving the system with 0 running requests and N waiting requests indefinitely. Removing this condition is a critical fix to ensure that the scheduler always attempts to schedule waiting requests if there are free resources, which is correct behavior after preemption frees up KV cache blocks.

[gemini-code-assist]

Disabling pinned memory for large CPU tensor allocations (>10GB) is a crucial fix to prevent potential GPU out-of-memory errors. Allocating large amounts of pinned memory can exhaust GPU page-table space, leading to CUDA errors. This change makes the CPU offloading feature more robust, especially when dealing with very large context windows that require significant CPU RAM for the KV cache.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Prefix cache ignores prompt_embeds

In generate_block_hash_extra_keys you removed _gen_prompt_embeds_extra_hash_keys and now only include LoRA/MM/salt keys (lines 502‑541). Requests that supply prompt_embeds no longer contribute any per‑request data to the block hash; Request._all_token_ids is just zero placeholders when embeddings are used, so two different embedding payloads of the same length now hash identically. This causes the prefix cache to treat unrelated prompts as identical, reusing another user’s KV blocks and returning incorrect completions, and it breaks the existing tests that verify prompt‑embed hashing. Please reintroduce the prompt‑embed bytes (or another unique key) into the block hash for these requests so cached blocks remain correct.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

count_tokens calls undefined helper

The new AnthropicServingMessages.count_tokens method (lines 464‑505) invokes await self._get_prompt_messages(...), but neither this class nor its OpenAI serving base defines _get_prompt_messages anywhere in the repository (a global search returns only this call). The first request to the count‑tokens endpoint will therefore raise AttributeError: 'AnthropicServingMessages' object has no attribute '_get_prompt_messages', making the API unusable. Please reuse the existing _get_prompt_components logic or add the missing helper before exposing this endpoint.

Useful? React with 👍 / 👎.

[heheda12345]

Close as it seems to be ai-generated.