[Bug]: Launching multiple vLLM processes at the same time doesn't work well with vLLM's compile cache

[zou3519]

Your current environment

My makefile:

clean:
    rm -rf /tmp/torchinductor_rzou
    rm -rf ~/.cache/vllm/torch_compile_cache
    killall -9  "VLLM::EngineCore"

run:
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=2 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=3 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=4 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=5 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=6 vllm serve --gpu-memory-utilization 0.1 &
    VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=7 vllm serve --gpu-memory-utilization 0.1 &

make run

🐛 Describe the bug

The problem is:

• we launch vllm in multiple processes
• each process tries to compile and save an artifact to disk in the same location
• each artifact is a directory
• the directories get clobbered

There's multiple ways I think we can fix the problem. I've tried some of these and haven't gotten them to work yet.

• standalone_compile offers a "binary" option instead of a "directory" option. writing to these binary files is atomic (we implemented it as a write+rename), reading from them is multiprocess-safe too (a process will not delete a file that's open until it is closed).
• if we go with the "directory" option, then we need some sort of read-write lock for CompiledArtifact.{save, load}. vLLM will always acquire said locks when it is performing save/load, and we can probably use something like a filelock for this. I've heard that filelocks are unreliable over nfs or other network stores, which does seem like how we are recommending vLLM users mount their ~/.cache/vllm directory, so I'm not sure how viable this is. We also have requirements like sometimes the vLLM cache directory is on a read-only filesystem, so we need to be cognizant of those too...

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[vadimkantorov]

I think the most important is to have safe-by-default behavior (e.g. maybe for directory option - always specify in-there some global rank or guid of the run). Of course best would be to have both safe (no data races) and performant (reuse of artifacts between runs/nodes via networked file-system or S3, s3 is not uncommon in small-cluster deployments)...

Also, one useful way could be being more explicit:

• have a vllm cache Qwen/Qwen3-0.6B command (with same args as vllm serve) which should populate the cache without any races and also download the model/tokenizer explicitly
• have other commands/vllm.LLM mode do only reads from cache, no writes. in this "fully cached" mode could also have HF_HUB_OFFLINE=1 to prevent hitting HF servers with HTTP requests

[zou3519]

cc @tlrmchlsmth @youkaichao @oulgen @jamesjwu @zhxchen17

From offline discussion with the vLLM folks it sounded like there were concerns with filelock (in that some distributed filesystems don't have good support for filelock).

One alternative is that standalone_compile can emit a "binary" (instead of a directory). The main issue with this I think is understandability (in the "directory" mode for standalone_compile, one can view and put breakpoints into Inductor's output_code file, but you cannot do this in the "binary" mode.

[jamesjwu]

What does it mean for files to be clobbered in this context? If VLLM's "cache key", i.e. directory where you're storing these artifacts is tight enough, then we should be writing the exact same files to the directory each time, no? In which case, a race condition shouldn't matter.

If there are different cache results saved to the same file, should we instead make sure that these cache results be placed in different directories based on some more stable key?

[vadimkantorov]

Could the directory mode also use some sort of rename trick? (as the single-file binary mode uses)

[oulgen]

Could the directory mode also use some sort of rename trick? (as the single-file binary mode uses)

unix does not support renaming of directories into a non-empty directory atomically

[zou3519]

@jamesjwu vLLM is supposed to be writing the exact same files to the directory each time.

I think there's two problems:

• autotuning: we do autotuning at compile time, so I'm not sure if we actually do get the same files if autotuning decisions change due to perf differences.
• CompiledArtifact.save(path) doesn't know if the same artifact is being saved to the path. So the two options for behavior are "override" or "don't save if a folder exists already". The current behavior is "override", and this involves a deletion for the folder. The act of doing the deletion while multiple processes do writes to the same folder leads to issues.

[vadimkantorov]

Does it support creating symlinks atomically? Maybe that could work around? Or maybe place all files in binary mode, and then symlink/hardlink these files in some dir for interpretability

Also, what do you think of a separate command/function for warmup (and a no-write, no-http request regime for LLM engine)? At least, it should allow pre-warm the cache - could be semantically meaningful thing even outside of the cache races problems

Also, could files be somehow content-addressable, e.g. include contents hash in the file name, so that autotuning differences do not write to the same file?

[zou3519]

I think we do want a separate vllm compile command. The guarantee would be that if you do:

vllm compile <options>
vllm serve <options> --error-on-compile

Then this guarantees vllm serve will no longer compile anything. This is a more explicit user model to work with.

That could also solve this problem in the way that we tell the user to vllm compile <options> and then they can launch as many processes as they want via vllm serve <options> --error-on-compile.

Also cc @ProExpertProg maybe I should write this into an RFC

[vadimkantorov]

For vllm serve and vllm.LLM, I'm also a bit concerned that it by default sends HTTP requests to HF all the time (reporting the model used etc), even if the model is downloaded (and if HF is overloaded it can lead to crashes - experienced HTTP429 crash myself). Would it make sense to at least make a public/arvertised argument/option for local_files_only?

E.g. I would imagine the user specifically calling vllm compile or maybe some new vllm cache as a prior step for downloading the model first

---

For vllm compile - I hope that it can still support writing to a networked file system directory from different machines (including with different autotune settings or different hardware - e.g. H100 and legacy A100). And hopefully it could also support reading/writing from S3 - maybe via fsspec

[ProExpertProg]

I think an RFC would likely be good here. We could also add an option where the cache uses binary files by default but we can add a utility for "decoding", as well as a flag to enable writing in plain text for dev users.