ZK-AI
/
livetalking
mirror of https://gitee.com/dumengxue/livetalking.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
livetalking/llm
History
yanyuxiyangzk@126.com 1c8f9338bb vllm文档 2024-04-03 15:50:59 +08:00
..
ChatGPT.py 加入LLM问答，如ChatGPT 2024-01-27 19:38:13 +08:00
Gemini.py 加入LLM问答，如ChatGPT 2024-01-27 19:38:13 +08:00
LLM.py 加入LLM问答，如ChatGPT 2024-01-27 19:38:13 +08:00
Qwen.py 加入LLM问答，如ChatGPT 2024-01-27 19:38:13 +08:00
README.md vllm文档 2024-04-03 15:50:59 +08:00

README.md

1、推理加速 conda create -n vllm python=3.10 conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia

python -m vllm.entrypoints.openai.api_server --tensor-parallel-size=1 --trust-remote-code --max-model-len 1024 --model THUDM/chatglm3-6b

python -m vllm.entrypoints.openai.api_server --host 127.0.0.1 --port 8101 --tensor-parallel-size=1 --trust-remote-code --max-model-len 1024 --model THUDM/chatglm3-6b