Problem with using text-generation-inference newer then version 1.1.1 on Openshift

[ptrmayer]

Hi all,
after deploying text-generation-inference images newer then version 1.1.1 and then running in the pod the following command:

$ text-generation-launcher

I get the following error:

2024-02-01T16:12:58.570827Z  INFO text_generation_launcher: Args { model_id: "bigscience/bloom-560m", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "rag-tgi-6cfdbf467-nvsp7", port: 8000, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, tokenizer_config_path: None, env: false }
2024-02-01T16:12:58.570866Z  INFO text_generation_launcher: Sharding model on 4 processes
2024-02-01T16:12:58.571047Z  INFO download: text_generation_launcher: Starting download process.
2024-02-01T16:13:01.674616Z ERROR download: text_generation_launcher: Download process was signaled to shutdown with signal 6: 
OMP: Error #179: Function Can't open SHM2 failed:
OMP: System error #13: Permission denied
Error: DownloadError

We are using Openshift 4.13.27.
Please let me know, if any other information from my side is required.
Very grateful for any hints concerning the cause of this issue.
Thanks!

[Narsil]

You should follow the error message, I don't think the error emanates from TGI itself (otherwise it would be log), but it seems to be instead the pod/host yelling.

the docker container will by default rty to write on /data to fetch the files for the model, that' s probably the issue here.