Internal CUDA allocator assertion failure during VAE encode in ComfyUI #10665
Description
Custom Node Testing
- I have tried disabling custom nodes and the issue persists.
Expected Behavior
The VAE encoding process in ComfyUI should execute successfully during tiled encoding on an NVIDIA RTX 2060 GPU (6GB VRAM) using PyTorch 2.8.0+cu129 with CUDA 12.9, without running into CUDA memory allocator assertion failures. The process should handle tiled image inputs with overlap and temporal settings gracefully, avoiding crashes, memory leaks, or assertion failures.
Actual Behavior
During tiled VAE encoding (VAEEncodeTiled node), ComfyUI crashes with the following error:
free_upper_bound + pytorch_used_bytes[device] <= device_total
INTERNAL ASSERT FAILED at "CUDAMallocAsyncAllocator.cpp":563, please report a bug to PyTorch.
This error consistently occurs during convolution operations in the encoding step, causing ComfyUI to quit unexpectedly. The crash happens despite reducing tile size and overlap parameters, enabling mixed precision (torch.amp.autocast), and avoiding unsafe cache clearing calls.
Steps to Reproduce
- Set up ComfyUI with models (Stable Diffusion and VAE) running on RTX 2060 with 6GB VRAM and NVIDIA driver 576.02.
- Install PyTorch 2.8.0+cu129 configured for CUDA 12.9.
- Enable VAE tiled encoding node with typical parameters (tile size 256-512, overlap 32-64, temporal size 64).
- Enable mixed precision in autoencoder encoding function using torch.amp.autocast.
- Attempt to run inference on images with resolution around 1024x2176 or larger.
- Observe crash logged with allocator assertion failure at CUDAMallocAsyncAllocator.cpp:563
Debug Logs
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 496, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 315, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 289, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 277, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\node_wrappers\leres.py", line 23, in execute
out = common_annotator_call(model, image, resolution=resolution, thr_a=rm_nearest, thr_b=rm_background, boost=boost == "enable")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\utils.py", line 85, in common_annotator_call
np_result = model(np_image, output_type="np", detect_resolution=detect_resolution, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\__init__.py", line 54, in __call__
depth = estimateboost(detected_map, self.model, 0, self.pix2pixmodel, max(detected_map.shape[1], detected_map.shape[0]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 420, in estimateboost
whole_estimate = doubleestimate(img, net_receptive_field_size, whole_image_optimal_size, pix2pixsize, model, model_type, pix2pixmodel)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 134, in doubleestimate
estimate1 = singleestimate(img, size1, model, net_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 158, in singleestimate
return estimateleres(img, model, msize, msize)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 45, in estimateleres
prediction = model.depth_model(img_torch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\multi_depth_model_woauxi.py", line 33, in forward
lateral_out = self.encoder_modules(x)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\network_auxi.py", line 96, in forward
x = self.encoder(x) # 1/32, 1/16, 1/8, 1/4
^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\Resnext_torch.py", line 223, in forward
return self._forward_impl(x)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\Resnext_torch.py", line 199, in _forward_impl
x = self.conv1(x)
^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\conv.py", line 548, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\conv.py", line 543, in _conv_forward
return F.conv2d(
^^^^^^^^^
System Information
- ComfyUI Version: 0.3.57
- Arguments: main.py
- OS: nt
- Python Version: 3.11.14 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 18:30:03) [MSC v.1929 64 bit (AMD64)]
- Embedded Python: false
- PyTorch Version: 2.8.0+cu129
Devices
- Name: cuda NVIDIA GeForce RTX 2060 : cudaMallocAsync
- Type: cuda
- VRAM Total: 6442123264
- VRAM Free: 5355077632
- Torch VRAM Total: 0
- Torch VRAM Free: 0
Logs
2025-11-06T08:43:26.499887 - [START] Security scan2025-11-06T08:43:26.499887 -
2025-11-06T08:43:31.109485 - [DONE] Security scan2025-11-06T08:43:31.109485 -
2025-11-06T08:43:31.579501 - ## ComfyUI-Manager: installing dependencies done.2025-11-06T08:43:31.579501 -
2025-11-06T08:43:31.579501 - ** ComfyUI startup time:2025-11-06T08:43:31.579501 - 2025-11-06T08:43:31.579501 - 2025-11-06 08:43:31.5792025-11-06T08:43:31.579501 -
2025-11-06T08:43:31.579501 - ** Platform:2025-11-06T08:43:31.579501 - 2025-11-06T08:43:31.579501 - Windows2025-11-06T08:43:31.579501 -
2025-11-06T08:43:31.579501 - ** Python version:2025-11-06T08:43:31.579501 - 2025-11-06T08:43:31.579501 - 3.11.14 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 18:30:03) [MSC v.1929 64 bit (AMD64)]2025-11-06T08:43:31.579501 -
2025-11-06T08:43:31.579501 - ** Python executable:2025-11-06T08:43:31.579501 - 2025-11-06T08:43:31.595537 - C:\Users\renat\Documents\ComfyUI\env\python.exe2025-11-06T08:43:31.595537 -
2025-11-06T08:43:31.652033 - ** ComfyUI Path:2025-11-06T08:43:31.652353 - 2025-11-06T08:43:31.652353 - C:\Users\renat\Documents\ComfyUI2025-11-06T08:43:31.653263 -
2025-11-06T08:43:31.653263 - ** ComfyUI Base Folder Path:2025-11-06T08:43:31.653263 - 2025-11-06T08:43:31.653263 - C:\Users\renat\Documents\ComfyUI2025-11-06T08:43:31.653263 -
2025-11-06T08:43:31.653263 - ** User directory:2025-11-06T08:43:31.653263 - 2025-11-06T08:43:31.653263 - C:\Users\renat\Documents\ComfyUI\user2025-11-06T08:43:31.653263 -
2025-11-06T08:43:31.653263 - ** ComfyUI-Manager config path:2025-11-06T08:43:31.653263 - 2025-11-06T08:43:31.653263 - C:\Users\renat\Documents\ComfyUI\user\default\ComfyUI-Manager\config.ini2025-11-06T08:43:31.653263 -
2025-11-06T08:43:31.653263 - ** Log path:2025-11-06T08:43:31.659319 - 2025-11-06T08:43:31.659319 - C:\Users\renat\Documents\ComfyUI\user\comfyui.log2025-11-06T08:43:31.659319 -
2025-11-06T08:43:36.452150 -
Prestartup times for custom nodes:
2025-11-06T08:43:36.452150 - 0.0 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\rgthree-comfy
2025-11-06T08:43:36.454171 - 13.5 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\ComfyUI-Manager
2025-11-06T08:43:36.454171 -
2025-11-06T08:43:43.005518 - Checkpoint files will always be loaded safely.
2025-11-06T08:43:43.383337 - Total VRAM 6144 MB, total RAM 16295 MB
2025-11-06T08:43:43.383337 - pytorch version: 2.8.0+cu129
2025-11-06T08:43:43.387357 - Set vram state to: NORMAL_VRAM
2025-11-06T08:43:43.389365 - Device: cuda NVIDIA GeForce RTX 2060 : cudaMallocAsync
2025-11-06T08:43:48.470785 - Using pytorch attention
2025-11-06T08:44:00.454242 - Python version: 3.11.14 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 18:30:03) [MSC v.1929 64 bit (AMD64)]
2025-11-06T08:44:00.454242 - ComfyUI version: 0.3.57
2025-11-06T08:44:00.539532 - ComfyUI frontend version: 1.25.11
2025-11-06T08:44:00.548617 - [Prompt Server] web root: C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\comfyui_frontend_package\static
2025-11-06T08:44:06.229109 - ### Loading: ComfyUI-Manager (V3.37)
2025-11-06T08:44:06.229109 - [ComfyUI-Manager] network_mode: public
2025-11-06T08:44:06.229109 - ### ComfyUI Revision: UNKNOWN (The currently installed ComfyUI is not a Git repository)
2025-11-06T08:44:06.436979 - �[36;20m[C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux] | INFO -> Using ckpts path: C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts�[0m
2025-11-06T08:44:06.436979 - �[36;20m[C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux] | INFO -> Using symlinks: False�[0m
2025-11-06T08:44:06.436979 - �[36;20m[C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider']�[0m
2025-11-06T08:44:06.495309 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
2025-11-06T08:44:06.573184 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
2025-11-06T08:44:06.718618 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
2025-11-06T08:44:06.968482 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
2025-11-06T08:44:07.244201 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
2025-11-06T08:44:09.468581 - �[92m[tinyterraNodes] �[32mLoaded�[0m2025-11-06T08:44:09.468581 -
2025-11-06T08:44:10.407949 - FETCH ComfyRegistry Data: 5/1042025-11-06T08:44:10.421602 -
2025-11-06T08:44:10.425636 -
2025-11-06T08:44:10.425636 - �[92m[rgthree-comfy] Loaded 48 exciting nodes. 🎉�[0m2025-11-06T08:44:10.425636 -
2025-11-06T08:44:10.425636 -
2025-11-06T08:44:13.854514 - FETCH ComfyRegistry Data: 10/1042025-11-06T08:44:13.855519 -
2025-11-06T08:44:15.529995 - �[34mWAS Node Suite: �[0mOpenCV Python FFMPEG support is enabled�[0m2025-11-06T08:44:15.529995 -
2025-11-06T08:44:15.529995 - �[34mWAS Node Suite �[93mWarning: �[0m`ffmpeg_bin_path` is not set in `C:\Users\renat\Documents\ComfyUI\custom_nodes\was-node-suite-comfyui\was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.�[0m2025-11-06T08:44:15.529995 -
2025-11-06T08:44:17.628540 - �[34mWAS Node Suite: �[0mFinished.�[0m �[32mLoaded�[0m �[0m220�[0m �[32mnodes successfully.�[0m2025-11-06T08:44:17.628540 -
2025-11-06T08:44:17.628540 -
�[3m�[93m"Art is the lie that enables us to recognize the truth."�[0m�[3m - Friedrich Nietzsche�[0m
2025-11-06T08:44:17.628540 -
2025-11-06T08:44:17.683331 -
Import times for custom nodes:
2025-11-06T08:44:17.683331 - 0.0 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\websocket_image_save.py
2025-11-06T08:44:17.683331 - 0.0 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\image-resize-comfyui
2025-11-06T08:44:17.683331 - 0.0 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\masquerade-nodes-comfyui
2025-11-06T08:44:17.683331 - 0.0 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_ipadapter_plus
2025-11-06T08:44:17.683331 - FETCH ComfyRegistry Data: 15/1042025-11-06T08:44:17.683331 - 0.0 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\jameswalker-nodes
2025-11-06T08:44:17.683331 -
2025-11-06T08:44:17.694317 - 0.1 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\Derfuu_ComfyUI_ModdedNodes
2025-11-06T08:44:17.695038 - 0.1 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\ComfyUI_UltimateSDUpscale
2025-11-06T08:44:17.695038 - 0.1 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_essentials
2025-11-06T08:44:17.695038 - 0.3 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\ComfyUI-Manager
2025-11-06T08:44:17.697061 - 0.3 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\ComfyUI-Image-Resizing
2025-11-06T08:44:17.698540 - 0.4 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_tinyterranodes
2025-11-06T08:44:17.699409 - 0.4 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\rgthree-comfy
2025-11-06T08:44:17.699409 - 2.9 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux
2025-11-06T08:44:17.701429 - 7.2 seconds: C:\Users\renat\Documents\ComfyUI\custom_nodes\was-node-suite-comfyui
2025-11-06T08:44:17.701429 -
2025-11-06T08:44:19.344720 - Context impl SQLiteImpl.
2025-11-06T08:44:19.344720 - Will assume non-transactional DDL.
2025-11-06T08:44:19.355123 - No target revision found.
2025-11-06T08:44:19.450851 - Starting server
2025-11-06T08:44:19.452850 - To see the GUI go to: http://127.0.0.1:8188
2025-11-06T08:44:21.161313 - FETCH ComfyRegistry Data: 20/1042025-11-06T08:44:21.161313 -
2025-11-06T08:44:24.667758 - FETCH ComfyRegistry Data: 25/1042025-11-06T08:44:24.667758 -
2025-11-06T08:44:28.119388 - FETCH ComfyRegistry Data: 30/1042025-11-06T08:44:28.119388 -
2025-11-06T08:44:31.603321 - FETCH ComfyRegistry Data: 35/1042025-11-06T08:44:31.603321 -
2025-11-06T08:44:35.705092 - FETCH ComfyRegistry Data: 40/1042025-11-06T08:44:35.705092 -
2025-11-06T08:44:39.179459 - FETCH ComfyRegistry Data: 45/1042025-11-06T08:44:39.179459 -
2025-11-06T08:44:42.626814 - FETCH ComfyRegistry Data: 50/1042025-11-06T08:44:42.626814 -
2025-11-06T08:44:46.112218 - FETCH ComfyRegistry Data: 55/1042025-11-06T08:44:46.112218 -
2025-11-06T08:44:49.599439 - FETCH ComfyRegistry Data: 60/1042025-11-06T08:44:49.599439 -
2025-11-06T08:44:53.083146 - FETCH ComfyRegistry Data: 65/1042025-11-06T08:44:53.083146 -
2025-11-06T08:44:56.608333 - FETCH ComfyRegistry Data: 70/1042025-11-06T08:44:56.608333 -
2025-11-06T08:45:00.108752 - FETCH ComfyRegistry Data: 75/1042025-11-06T08:45:00.108752 -
2025-11-06T08:45:03.603472 - FETCH ComfyRegistry Data: 80/1042025-11-06T08:45:03.603472 -
2025-11-06T08:45:07.170827 - FETCH ComfyRegistry Data: 85/1042025-11-06T08:45:07.172837 -
2025-11-06T08:45:11.369391 - FETCH ComfyRegistry Data: 90/1042025-11-06T08:45:11.369391 -
2025-11-06T08:45:14.840713 - FETCH ComfyRegistry Data: 95/1042025-11-06T08:45:14.840713 -
2025-11-06T08:45:18.905579 - FETCH ComfyRegistry Data: 100/1042025-11-06T08:45:18.905579 -
2025-11-06T08:45:22.238891 - FETCH ComfyRegistry Data [DONE]2025-11-06T08:45:22.238891 -
2025-11-06T08:45:23.285155 - [ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
2025-11-06T08:45:23.427590 - FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json2025-11-06T08:45:23.427590 - 2025-11-06T08:45:23.603429 - [DONE]2025-11-06T08:45:23.603429 -
2025-11-06T08:45:24.006841 - [ComfyUI-Manager] All startup tasks have been completed.
2025-11-06T08:46:44.363498 - got prompt
2025-11-06T08:46:44.435193 - Failed to validate prompt for output 170:
2025-11-06T08:46:44.436793 - * BasicScheduler 17:
2025-11-06T08:46:44.438437 - - Required input is missing: model
2025-11-06T08:46:44.438437 - * Text to Conditioning 171:
2025-11-06T08:46:44.440058 - - Required input is missing: clip
2025-11-06T08:46:44.440058 - * BasicGuider 22:
2025-11-06T08:46:44.441720 - - Required input is missing: model
2025-11-06T08:46:44.441720 - Output will be ignored
2025-11-06T08:46:44.443316 - Failed to validate prompt for output 48:
2025-11-06T08:46:44.444938 - * UltimateSDUpscale 177:
2025-11-06T08:46:44.446665 - - Required input is missing: image
2025-11-06T08:46:44.446665 - - Required input is missing: upscale_model
2025-11-06T08:46:44.448243 - Output will be ignored
2025-11-06T08:46:44.449831 - Failed to validate prompt for output 149:
2025-11-06T08:46:44.451214 - Output will be ignored
2025-11-06T08:46:44.452868 - Failed to validate prompt for output 164:
2025-11-06T08:46:44.454911 - * (prompt):
2025-11-06T08:46:44.454911 - - Required input is missing: images
2025-11-06T08:46:44.455940 - * PreviewImage 164:
2025-11-06T08:46:44.457530 - - Required input is missing: images
2025-11-06T08:46:44.457530 - Output will be ignored
2025-11-06T08:46:44.459084 - Failed to validate prompt for output 9:
2025-11-06T08:46:44.460682 - Output will be ignored
2025-11-06T08:46:44.641138 - WARNING: [Errno 2] No such file or directory: 'C:\\Users\\renat\\Documents\\ComfyUI\\input\\01_3 - Photo.jpg'
2025-11-06T08:47:29.247313 - model weight dtype torch.bfloat16, manual cast: torch.float32
2025-11-06T08:47:29.334276 - model_type EPS
2025-11-06T08:47:34.956631 - Using pytorch attention in VAE
2025-11-06T08:47:34.964443 - Using pytorch attention in VAE
2025-11-06T08:47:36.834924 - VAE load device: cuda, offload device: cuda, dtype: torch.float32
2025-11-06T08:47:37.839063 - Requested to load SDXLClipModel
2025-11-06T08:47:37.994763 - loaded completely 9.5367431640625e+25 1560.802734375 True
2025-11-06T08:47:38.036036 - CLIP/text encoder model load device: cuda, offload device: cuda, current: cuda, dtype: torch.float16
2025-11-06T08:47:38.673031 - model_path is C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts\lllyasviel\Annotators\res101.pth2025-11-06T08:47:38.673031 -
2025-11-06T08:47:47.433462 - model_path is C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts\lllyasviel\Annotators\latest_net_G.pth2025-11-06T08:47:47.433462 -
2025-11-06T08:47:59.465894 - !!! Exception during processing !!! free_upper_bound + pytorch_used_bytes[device] <= device_total INTERNAL ASSERT FAILED at "C:\\actions-runner\\_work\\pytorch\\pytorch\\pytorch\\c10\\cuda\\CUDAMallocAsyncAllocator.cpp":563, please report a bug to PyTorch.
2025-11-06T08:47:59.489176 - Traceback (most recent call last):
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 496, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 315, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 289, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "C:\Users\renat\Documents\ComfyUI\execution.py", line 277, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\node_wrappers\leres.py", line 23, in execute
out = common_annotator_call(model, image, resolution=resolution, thr_a=rm_nearest, thr_b=rm_background, boost=boost == "enable")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\utils.py", line 85, in common_annotator_call
np_result = model(np_image, output_type="np", detect_resolution=detect_resolution, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\__init__.py", line 54, in __call__
depth = estimateboost(detected_map, self.model, 0, self.pix2pixmodel, max(detected_map.shape[1], detected_map.shape[0]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 420, in estimateboost
whole_estimate = doubleestimate(img, net_receptive_field_size, whole_image_optimal_size, pix2pixsize, model, model_type, pix2pixmodel)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 134, in doubleestimate
estimate1 = singleestimate(img, size1, model, net_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 158, in singleestimate
return estimateleres(img, model, msize, msize)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\depthmap.py", line 45, in estimateleres
prediction = model.depth_model(img_torch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\multi_depth_model_woauxi.py", line 33, in forward
lateral_out = self.encoder_modules(x)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\network_auxi.py", line 96, in forward
x = self.encoder(x) # 1/32, 1/16, 1/8, 1/4
^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\Resnext_torch.py", line 223, in forward
return self._forward_impl(x)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_controlnet_aux\leres\leres\Resnext_torch.py", line 199, in _forward_impl
x = self.conv1(x)
^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\conv.py", line 548, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\renat\Documents\ComfyUI\env\Lib\site-packages\torch\nn\modules\conv.py", line 543, in _conv_forward
return F.conv2d(
^^^^^^^^^
RuntimeError: free_upper_bound + pytorch_used_bytes[device] <= device_total INTERNAL ASSERT FAILED at "C:\\actions-runner\\_work\\pytorch\\pytorch\\pytorch\\c10\\cuda\\CUDAMallocAsyncAllocator.cpp":563, please report a bug to PyTorch.
2025-11-06T08:47:59.504747 - Prompt executed in 75.02 seconds
Other
- VRAM: 6 GB
- OS: Windows 11
- Environment: Python 3.11, PyTorch 2.8.0+cu129
- Attempted mitigation: Reduced tile and overlap sizes, enabled mixed precision wrapping, used environment variable PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:64.
- Avoided clearing cache (torch.cuda.empty_cache()) due to previous ComfyUI crashes.
- Issue appears linked to memory fragmentation or CUDA allocator behavior under tiled (or default) VAE loads.