vllm serve /home/fgg/modelscope/Qwen/Qwen2-7b --tensor-parallel-size 2
INFO 12-15 16:17:56 [__init__.py:256] Automatically detected platform cuda.
INFO 12-15 16:17:57 [api_server.py:977] vLLM API server version 0.8.0
INFO 12-15 16:17:57 [api_server.py:978] args: Namespace(subparser='serve', model_tag='/home/fgg/modelscope/Qwen/Qwen2-7b', config='', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/home/fgg/modelscope/Qwen/Qwen2-7b', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, use_tqdm_on_load=True, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7dcce187c360>)
`torch_dtype` is deprecated! Use `dtype` instead!
INFO 12-15 16:17:59 [config.py:583] This model supports multiple tasks: {'embed', 'generate', 'classify', 'score', 'reward'}. Defaulting to 'generate'.
INFO 12-15 16:17:59 [config.py:1515] Defaulting to use mp for distributed inference
INFO 12-15 16:17:59 [config.py:1693] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 12-15 16:18:02 [__init__.py:256] Automatically detected platform cuda.
INFO 12-15 16:18:03 [core.py:53] Initializing a V1 LLM engine (v0.8.0) with config: model='/home/fgg/modelscope/Qwen/Qwen2-7b', speculative_config=None, tokenizer='/home/fgg/modelscope/Qwen/Qwen2-7b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/home/fgg/modelscope/Qwen/Qwen2-7b, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 12-15 16:18:03 [multiproc_worker_utils.py:310] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 12-15 16:18:03 [custom_cache_manager.py:19] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 12-15 16:18:03 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 10485760, 10, 'psm_a8228524'), local_subscribe_addr='ipc:///tmp/e8c2ea83-b990-4e5c-b130-14727f8446ea', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 12-15 16:18:04 [__init__.py:256] Automatically detected platform cuda.
/home/fgg/anaconda3/envs/vllm/lib/python3.12/site-packages/tvm_ffi/_optional_torch_c_dlpack.py:174: UserWarning: Failed to JIT torch c dlpack extension, EnvTensorAllocator will not be enabled.
We recommend installing via `pip install torch-c-dlpack-ext`
warnings.warn(
WARNING 12-15 16:18:05 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7d9b7df17740>
(VllmWorker rank=0 pid=3421367) INFO 12-15 16:18:05 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_c51d43cf'), local_subscribe_addr='ipc:///tmp/88574035-8b98-4161-a458-4490fcd4e6f6', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 12-15 16:18:07 [__init__.py:256] Automatically detected platform cuda.
/home/fgg/anaconda3/envs/vllm/lib/python3.12/site-packages/tvm_ffi/_optional_torch_c_dlpack.py:174: UserWarning: Failed to JIT torch c dlpack extension, EnvTensorAllocator will not be enabled.
We recommend installing via `pip install torch-c-dlpack-ext`
warnings.warn(
WARNING 12-15 16:18:08 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7340826a6c60>
(VllmWorker rank=1 pid=3421398) INFO 12-15 16:18:08 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_098a6598'), local_subscribe_addr='ipc:///tmp/61ebe5ab-789e-4f42-b06b-d0f35704546f', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=3421367) INFO 12-15 16:18:08 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=3421398) INFO 12-15 16:18:08 [utils.py:925] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=3421367) INFO 12-15 16:18:08 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=1 pid=3421398) INFO 12-15 16:18:08 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=0 pid=3421367) INFO 12-15 16:18:08 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /home/fgg/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorker rank=1 pid=3421398) INFO 12-15 16:18:08 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /home/fgg/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorker rank=0 pid=3421367) WARNING 12-15 16:18:08 [custom_all_reduce.py:146] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=1 pid=3421398) WARNING 12-15 16:18:08 [custom_all_reduce.py:146] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=0 pid=3421367) INFO 12-15 16:18:08 [shm_broadcast.py:258] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_7da022e6'), local_subscribe_addr='ipc:///tmp/677d6179-a939-4c2a-945d-e79314a6fa80', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=1 pid=3421398) INFO 12-15 16:18:08 [parallel_state.py:967] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1
(VllmWorker rank=0 pid=3421367) INFO 12-15 16:18:08 [parallel_state.py:967] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0
(VllmWorker rank=0 pid=3421367) Process SpawnProcess-1:1:
(VllmWorker rank=1 pid=3421398) Process SpawnProcess-1:2:
CRITICAL 12-15 16:18:08 [multiproc_executor.py:48] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
CRITICAL 12-15 16:18:08 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
已杀死
最新发布
最低0.47元/天 解锁文章
扫一扫
4万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
