torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 214.00 MiB. GPU 0 has a total capacity

本地有两张GPU，NVIDIA GeForce RTX 4080 和 NVIDIA GeForce RTX 4080 SUPER但是

启动报错

oading checkpoint shards: 80%|████████████████████████████████████████████████████████████████████████████▊ | 8/10 [01:38<00:24, 12.30s/it] 2024-11-16 08:46:26.929 Uncaught app exception Traceback (most recent call last): File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 575, in _run_script self._session_state.on_script_will_rerun( File "/home/xx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/state/safe_session_state.py", line 65, in on_script_will_rerun self._state.on_script_will_rerun(latest_widget_states) File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/state/session_state.py", line 517, in on_script_will_rerun self._call_callbacks() File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/state/session_state.py", line 530, in _call_callbacks self._new_widget_state.call_callback(wid) File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/state/session_state.py", line 274, in call_callback callback(*args, **kwargs) File "/server/repo/GLM-4/composite_demo/src/main.py", line 130, in page_changed st.session_state.client = build_client(Mode(new_page)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/server/repo/GLM-4/composite_demo/src/main.py", line 114, in build_client return get_client(CHAT_MODEL_PATH, typ) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/caching/cache_utils.py", line 165, in wrapper return cached_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/caching/cache_utils.py", line 194, in __call__ return self._get_or_create_cached_value(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/caching/cache_utils.py", line 221, in _get_or_create_cached_value return self._handle_cache_miss(cache, value_key, func_args, func_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/streamlit/runtime/caching/cache_utils.py", line 277, in _handle_cache_miss computed_value = self._info.func(*func_args, **func_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/server/repo/GLM-4/composite_demo/src/client.py", line 89, in get_client return HFClient(model_path) ^^^^^^^^^^^^^^^^^^^^ File "/server/repo/GLM-4/composite_demo/src/clients/hf.py", line 21, in __init__ self.model = AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4225, in from_pretrained ) = cls._load_pretrained_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4728, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xxx/anaconda3/envs/glm-4-demo/lib/python3.12/site-packages/transformers/modeling_utils.py", line