122 lines
3.4 KiB
Markdown
122 lines
3.4 KiB
Markdown
# 虚拟人说话头生成(照片虚拟人实时驱动)
|
||
|
||
# Get Started
|
||
|
||
## Installation
|
||
|
||
Tested on Ubuntu 22.04, Pytorch 1.12 and CUDA 11.6，or Pytorch 1.12 and CUDA 11.3
|
||
|
||
```python
|
||
git clone https://github.com/waityousea/xuniren.git
|
||
cd xuniren
|
||
```
|
||
|
||
### Install dependency
|
||
|
||
```python
|
||
# for ubuntu, portaudio is needed for pyaudio to work.
|
||
sudo apt install portaudio19-dev
|
||
|
||
pip install -r requirements.txt
|
||
or
|
||
## environment.yml中的pytorch使用的1.12和cuda 11.3
|
||
conda env create -f environment.yml
|
||
## install pytorch3d
|
||
#ubuntu/mac
|
||
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
|
||
```
|
||
|
||
**windows安装pytorch3d**
|
||
|
||
- gcc & g++ ≥ 4.9
|
||
|
||
在windows中，需要安装gcc编译器，可以根据需求自行安装，例如采用MinGW
|
||
|
||
以下安装步骤来自于[pytorch3d](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md)官方, 可以根据需求进行选择。
|
||
|
||
```python
|
||
conda create -n pytorch3d python=3.9
|
||
conda activate pytorch3d
|
||
conda install pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
|
||
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
|
||
```
|
||
|
||
对于 CUB 构建时间依赖项，仅当您的 CUDA 早于 11.7 时才需要，如果您使用的是 conda，则可以继续
|
||
|
||
```
|
||
conda install -c bottler nvidiacub
|
||
```
|
||
|
||
```
|
||
# Demos and examples
|
||
conda install jupyter
|
||
pip install scikit-image matplotlib imageio plotly opencv-python
|
||
|
||
# Tests/Linting
|
||
pip install black usort flake8 flake8-bugbear flake8-comprehensions
|
||
```
|
||
|
||
任何必要的补丁后，你可以去“x64 Native Tools Command Prompt for VS 2019”编译安装
|
||
|
||
```
|
||
git clone https://github.com/facebookresearch/pytorch3d.git
|
||
cd pytorch3d
|
||
python setup.py install
|
||
```
|
||
|
||
### Build extension
|
||
|
||
By default, we use [`load`](https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load) to build the extension at runtime. However, this may be inconvenient sometimes. Therefore, we also provide the `setup.py` to build each extension:
|
||
|
||
```
|
||
# install all extension modules
|
||
# notice: 该模块必须安装。
|
||
# 在windows下，建议采用vs2019的x64 Native Tools Command Prompt for VS 2019命令窗口安装
|
||
bash scripts/install_ext.sh
|
||
```
|
||
|
||
### **start(独立运行)**
|
||
|
||
环境配置完成后，启动虚拟人生成器：
|
||
|
||
```python
|
||
python app.py
|
||
```
|
||
### **start（对接fay，在ubuntu 20下完成测试）**
|
||
环境配置完成后，启动fay对接脚本
|
||
```python
|
||
python fay_connect.py
|
||
```
|
||
|
||
|
||
扫码支助开源开发工作，凭支付单号入qq交流群
|
||
|
||
|
||
|
||
接口的输入与输出信息 [Websoket.md](https://github.com/waityousea/xuniren/blob/main/WebSocket.md)
|
||
|
||
虚拟人生成的核心文件
|
||
|
||
```python
|
||
## 注意，核心文件需要单独训练
|
||
.
|
||
├── data
|
||
│ ├── kf.json
|
||
│ ├── pretrained
|
||
│ └── └── ngp_kg.pth
|
||
|
||
```
|
||
|
||
### Inference Speed
|
||
|
||
在台式机RTX A4000或笔记本RTX 3080ti的显卡（显存16G）上进行视频推理时，1s可以推理35~43帧，假如1s视频25帧，则1s可推理约1.5s视频。
|
||
|
||
# Acknowledgement
|
||
|
||
- The data pre-processing part is adapted from [AD-NeRF](https://github.com/YudongGuo/AD-NeRF).
|
||
- The NeRF framework is based on [torch-ngp](https://github.com/ashawkey/torch-ngp).
|
||
- The algorithm core come from [RAD-NeRF](https://github.com/ashawkey/RAD-NeRF).
|
||
- Usage example [Fay](https://github.com/TheRamU/Fay).
|
||
|
||
学术交流可发邮件到邮箱：waityousea@126.com
|