runtime error

Exit code: 1. Reason: ageTransformer2DModel'>. Try_On_Qwen_Edit_Lora.safetensors: 0%| | 0.00/590M [00:00<?, ?B/s] Try_On_Qwen_Edit_Lora.safetensors: 0%| | 0.00/590M [00:01<?, ?B/s] Try_On_Qwen_Edit_Lora.safetensors: 11%|█▏ | 67.2M/590M [00:02<00:09, 56.1MB/s] Try_On_Qwen_Edit_Lora.safetensors: 100%|██████████| 590M/590M [00:02<00:00, 226MB/s] SPACES_ZERO_GPU_DEBUG self.arg_queue._writer.fileno()=6 SPACES_ZERO_GPU_DEBUG self.res_queue._writer.fileno()=8 Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 149, in worker_init torch.move(callback=callback) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 447, in move e.submit(copy_context().run, _move, callback=callback).result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 430, in _move original_cuda = original.pin_memory().cuda(non_blocking=True) RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":1123, please report a bug to PyTorch. Traceback (most recent call last): File "/home/user/app/app.py", line 43, in <module> optimize_pipeline_(pipe, image=[Image.new("RGB", (1024, 1024)), Image.new("RGB", (1024, 1024))], prompt="prompt") File "/home/user/app/optimization.py", line 70, in optimize_pipeline_ spaces.aoti_apply(compile_transformer(), pipeline.transformer) File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 224, in gradio_handler raise error("ZeroGPU worker error", res.error_cls) gradio.exceptions.Error: 'RuntimeError'

Container logs: