WonderJourney 项目部署踩坑记录
WonderJourney 项目部署踩坑记录
废话不多说，正篇开始！！！
部署流程
项目地址： https://github.com/KovenYu/WonderJourney
my基础环境：
- pytorch= 2.0.1 ；
- python= 3.10.6；
- cuda= 11.7；
- torchvision =0.15.2+cu117；
项目环境部署
git clone https://github.com/KovenYu/WonderJourney.git
cd WonderJourney
conda create --name wonderjourney python=3.10
conda activate wonderjourney
Pytorch3D部署
借鉴该博主方法: https://blog.youkuaiyun.com/weixin_43891061/article/details/131211465
部署指令简单，部署方便: 
#可以提前配置pytorch3d依赖库
pip install fvcore iopath
#我的环境兼容的pytorch3d指令
conda install https://anaconda.org/pytorch3d/pytorch3d/0.7.4/download/linux-64/pytorch3d-0.7.8-py310_cu117_pyt201.tar.bz2
配置项目依赖库
关键在这，官方库版本描述太模糊，并且缺少依赖库指令，我将精确版本如下.
// av和openai
pip install --upgrade pip
pip install kornia
pip install matplotlib
pip install opencv-python
pip install segment-anything
pip install scikit-image
pip install diffusers
pip install transformers
pip install accelerate
timm==0.6.7
pillow==9.2.0
pip install einops
pip install omegaconf
pip install av==11.0.0
pip install openai
pip install ipdb
pip install spacy
python -m spacy download en_core_web_sm #加载语言模型
pip install urllib3==1.25.11
2个权重文件需手动下载，放到根目录：
https://huggingface.co/spaces/abhishek/StableSAM/blob/main/sam_vit_h_4b8939.pth
https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt
改代码util / segment_utils.py 文件的第2行（注释）和第67行（sam_vit_h_4b8939.pth路径）
至此基本环境部署完成，大部分问题是获取openai令牌
openai令牌（难点）
借鉴该博主方法: https://blog.youkuaiyun.com/m0_73153984/article/details/143744267
国内服务器无法访问外网，故采用国内第三方平台
方法如下：
例如 https://api.v3.cm
1、注册，创建令牌，记录此令牌，方便后续使用（控制台->我的令牌）
2、修改代码util / ChatGPT4.py
import openai
import json
import time
from pathlib import Path
import io
import base64
import requests
import spacy
import os
# run 'python -m spacy download en_core_web_sm' to load english language model
nlp = spacy.load("en_core_web_sm")
openai.api_key = os.environ['OPENAI_API_KEY']
from openai import OpenAI
client = OpenAI(
api_key="sk-xxxxxx", # OpenAI的密钥
base_url = 'https://api.v3.cm/v1' # 第三方接口或者是国外服务器，注意网址后要加/v1，否则会报错
)
class TextpromptGen(object):
def __init__(self, root_path, control=False):
super(TextpromptGen, self).__init__()
self.model = "gpt-4"
self.save_prompt = True
self.scene_num = 0
if control:
self.base_content = "Please generate scene description based on the given information:"
else:
self.base_content = "Please generate next scene based on the given scene/scenes information:"
self.content = self.base_content
self.root_path = root_path
def write_json(self, output, save_dir=None):
if save_dir is None:
save_dir = Path(self.root_path)
save_dir.mkdir(parents=True, exist_ok=True)
try:
output['background'][0] = self.generate_keywords(output['background'][0])
with open(save_dir / 'scene_{}.json'.format(str(self.scene_num).zfill(2)), "w") as json_file:
json.dump(output, json_file, indent=4)
except Exception as e:
pass
return
def write_all_content(self, save_dir=None):
if save_dir is None:
save_dir = Path(self.root_path)
save_dir.mkdir(parents=True, exist_ok=True)
with open(save_dir / 'all_content.txt', "w") as f:
f.write(self.content)
return
def regenerate_background(self, style, entities, scene_name, background=None):
if background is not None:
content = "Please generate a brief scene background with Scene name: " + scene_name + "; Background: " + str(background).strip(".") + ". Entities: " + str(entities) + "; Style: " + str(style)
else:
content = "Please generate a brief scene background with Scene name: " + scene_name + "; Entities: " + str(entities) + "; Style: " + str(style)
messages = [{"role": "system", "content": "You are an intelligent scene generator. Given a scene and there are 3 most significant common entities. please generate a brief background prompt about 50 words describing common things in the scene. You should not mention the entities in the background prompt. If needed, you can make reasonable guesses."}, \
{"role": "user", "content": content}]
response = client.chat.completions.create(
model=self.model,
messages=messages,
timeout=5,
)
# background = response['choices'][0]['message']['content']
background = response.choices[0].message.content
return background.strip(".")
def run_conversation(self, style=None, entities=None, scene_name=None, background=None, control_text=None):
######################################
# Input ------------------------------
# scene_name: str
# entities: List(str) ['entity_1', 'entity_2', 'entity_3']
# style: str
######################################
# Output -----------------------------
# output: dict {'scene_name': [''], 'entities': ['', '', ''], 'background': ['']}
if control_text is not None:
self.scene_num += 1
scene_content = "\n{Scene information: " + str(control_text).strip(".") + "; Style: " + str(style) + "}"
self.content = self.base_content + scene_content
elif style is not None and entities is not None:
assert not (background is None and scene_name is None), 'At least one of the background and scene_name should not be None'
self.scene_num += 1
if background is not None:
if isinstance(background, list):
background = background[0]
scene_content = "\nScene " + str(self.scene_num) + ": " + "{Background: " + str(background).strip(".") + ". Entities: " + str(entities) + "; Style: " + str(style) + "}"
else:
if isinstance(scene_name, list):
scene_name = scene_name[0]
scene_content = "\nScene " + str(self.scene_num) + ": " + "{Scene name: " + str(scene_name).strip(".") + "; Entities: " + str(entities) + "; Style: " + str(style) + "}"
self.content += scene_content
else:
assert self.scene_num > 0, 'To regenerate the scene description, you should have at least one scene content as prompt.'
if control_text is not None:
messages = [{"role": "system", "content": "You are an intelligent scene description generator. Given a sentence describing a scene, please translate it into English if not and summarize the scene name and 3 most significant common entities in the scene. You also have to generate a brief background prompt about 50 words describing the scene. You should not mention the entities in the background prompt. If needed, you can make reasonable guesses. Please use the format below: (the output should be json format)\n \
{'scene_name': ['scene_name'], 'entities': ['entity_1', 'entity_2', 'entity_3'], 'background': ['background prompt']}"}, \
{"role": "user", "content": self.content}]
else:
messages = [{"role": "system", "content": "You are an intelligent scene generator. Imaging you are flying through a scene or a sequence of scenes, and there are 3 most significant common entities in each scene. Please tell me what sequentially next scene would you likely to see? You need to generate the scene name and the 3 most common entities in the scene. The scenes are sequentially interconnected, and the entities within the scenes are adapted to match and fit with the scenes. You also have to generate a brief background prompt about 50 words describing the scene. You should not mention the entities in the background prompt. If needed, you can make reasonable guesses. Please use the format below: (the output should be json format)\n \
{'scene_name': ['scene_name'], 'entities': ['entity_1', 'entity_2', 'entity_3'], 'background': ['background prompt']}"}, \
{"role": "user", "content": self.content}]
for i in range(10):
try:
response = client.chat.completions.create(
model=self.model,
messages=messages,
timeout=5,
)
response = response.choices[0].message.content
try:
print(response)
output = eval(response)
_, _, _ = output['scene_name'], output['entities'], output['background']
if isinstance(output, tuple):
output = output[0]
if isinstance(output['scene_name'], str):
output['scene_name'] = [output['scene_name']]
if isinstance(output['entities'], str):
output['entities'] = [output['entities']]
if isinstance(output['background'], str):
output['background'] = [output['background']]
break
except Exception as e:
assistant_message = {"role": "assistant", "content": response}
user_message = {"role": "user", "content": "The output is not json format, please try again:\n" + self.content}
messages.append(assistant_message)
messages.append(user_message)
print("An error occurred when transfering the output of chatGPT into a dict, chatGPT4, let's try again!", str(e))
continue
except openai.APIError as e:
print(f"OpenAI API returned an API Error: {e}")
print("Wait for a second and ask chatGPT4 again!")
time.sleep(1)
continue
if self.save_prompt:
self.write_json(output)
return output
def generate_keywords(self, text):
doc = nlp(text)
adj = False
noun = False
text = ""
for token in doc:
if token.pos_ != "NOUN" and token.pos_ != "ADJ":
continue
if token.pos_ == "NOUN":
if adj:
text += (" " + token.text)
adj = False
noun = True
else:
if noun:
text += (", " + token.text)
else:
text += token.text
noun = True
elif token.pos_ == "ADJ":
if adj:
text += (" " + token.text)
else:
if noun:
text += (", " + token.text)
noun = False
adj = True
else:
text += token.text
adj = True
return text
def generate_prompt(self, style, entities, background=None, scene_name=None):
assert not (background is None and scene_name is None), 'At least one of the background and scene_name should not be None'
if background is not None:
if isinstance(background, list):
background = background[0]
background = self.generate_keywords(background)
prompt_text = "Style: " + style + ". Entities: "
for i, entity in enumerate(entities):
if i == 0:
prompt_text += entity
else:
prompt_text += (", " + entity)
prompt_text += (". Background: " + background)
print('PROMPT TEXT: ', prompt_text)
else:
if isinstance(scene_name, list):
scene_name = scene_name[0]
prompt_text = "Style: " + style + ". " + scene_name + " with "
for i, entity in enumerate(entities):
if i == 0:
prompt_text += entity
elif i == len(entities) - 1:
prompt_text += (", and " + entity)
else:
prompt_text += (", " + entity)
return prompt_text
def encode_image_pil(self, image):
with io.BytesIO() as buffer:
image.save(buffer, format='PNG')
return base64.b64encode(buffer.getvalue()).decode('utf-8')
def evaluate_image(self, image, eval_blur=True):
api_key = openai.api_key
base64_image = self.encode_image_pil(image)
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {openai.api_key}"
}
payload = {
"model": "gpt-4-vision-preview",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": ""
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
}
}
]
}
],
"max_tokens": 300
}
border_text = "Along the four borders of this image, is there anything that looks like thin border, thin stripe, photograph border, painting border, or painting frame? Please look very closely to the four edges and try hard, because the borders are very slim and you may easily overlook them. I would lose my job if there is a border and you overlook it. If you are not sure, then please say yes."
print(border_text)
has_border = True
payload['messages'][0]['content'][0]['text'] = border_text + " Your answer should be simply 'Yes' or 'No'."
for i in range(5):
try:
response = requests.post("https://api.v3.cm/v1/chat/completions", headers=headers, json=payload, timeout=5)
border = response.json()['choices'][0]['message']['content'].strip(' ').strip('.').lower()
if border in ['yes', 'no']:
print('Border: ', border)
has_border = border == 'yes'
break
except Exception as e:
print("Something has been wrong while asking GPT4V. Wait for a second and ask chatGPT4 again!")
time.sleep(1)
continue
if eval_blur:
blur_text = "Does this image have a significant blur issue or blurry effect caused by out of focus around the image edges? You only have to pay attention to the four borders of the image."
print(blur_text)
payload['messages'][0]['content'][0]['text'] = blur_text + " Your answer should be simply 'Yes' or 'No'."
for i in range(5):
try:
response = requests.post("https://api.v3.cm/v1/chat/completions", headers=headers, json=payload, timeout=5)
blur =response.json()['choices'][0]['message']['content'].strip(' ').strip('.').lower()
if blur in ['yes', 'no']:
print('Blur: ', blur)
break
except Exception as e:
print("Something has been wrong while asking GPT4V. Wait for a second and ask chatGPT4 again!")
time.sleep(1)
continue
has_blur = blur == 'yes'
else:
has_blur = False
openai.api_key = api_key
return has_border, has_blur
完成！
测试
可以运行指令测试啦
python run.py --example_config config/village.yaml
如果想测试自定义图片，可以按照如下：
- 添加新图像
./examples/images/。 - 添加此新图像的内容
./examples/examples.yaml - 编写一个
config/new_example.yaml类似于./config/village.yaml新示例的配置 - 运行
python run.py --example_config config/new_example.yaml
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
