Diffusers

最新推荐文章于 2025-03-11 21:41:34 发布

zxyzxyzxyzxyzz

最新推荐文章于 2025-03-11 21:41:34 发布

阅读量878

点赞数 8

文章标签： python

本文链接：https://blog.youkuaiyun.com/hhshbubaoyuan/article/details/145142651

版权

一、diffusers版本>=0.32.0时，模型量化问题

1、bitsandbytes，运行正常但是采样速度变慢

需要下载对应分支pip install git+https://github.com/BenjaminBossan/peft@fix-low-cpu-mem-usage-bnb-8bit

transformer_8bit = FluxTransformer2DModel.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    subfolder="transformer",
    quantization_config=DiffusersBitsAndBytesConfig(load_in_8bit=True),
    torch_dtype=torch.bfloat16,
)
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    transformer=transformer_8bit,
    torch_dtype=torch.bfloat16,
).to("cuda")

pipe.load_lora_weights(
    hf_hub_download("ByteDance/Hyper-SD", "Hyper-FLUX.1-dev-8steps-lora.safetensors"),
    adapter_name="hyper-sd"
)
pipe.set_adapters("hyper-sd", adapter_weights=0.125)

2、optimum，导入lora时存在问题，待解决

Traceback (most recent call last):
File "/home/iv/anaconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
File "/home/iv/anaconda3/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/app.py", line 112, in process_data
    IVmodelHandle.loadModel()
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/algos/BaseModel.py", line 28, in loadModel
    self.model = GlobalConfig.get_model()
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/algos/GlobalConfig.py", line 76, in get_model
    modelHandler = model()
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP/algos/FluxTxt2Img/Flux_txt2img.py", line 145, in __init__
    self.pipe.load_lora_weights(hf_hub_download("Shakker-Labs/FLUX.1-dev-LoRA-Logo-Design", "FLUX-dev-lora-Logo-Design.safetensors"), adapter_name="logo")
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/66v/lib/python3.10/site-packages/diffusers/loaders/lora_pipeline.py", line 1559, in load_lora_weights
    transformer_lora_state_dict = self._maybe_expand_lora_state_dict(
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/66v/lib/python3.10/site-packages/diffusers/loaders/lora_pipeline.py", line 2085, in _maybe_expand_lora_state_dict
    base_weight_param = transformer_state_dict[base_param_name]
KeyError: 'single_transformer_blocks.0.attn.to_k.weight._data'

二、某些lora加载失败

解决办法：diffusers版本过低，升级diffusers

Traceback (most recent call last):
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP-test/algos/AnystyleImg2Img/newfilter.py", line 1042, in <module>
    model = AnyStyleImg2ImgModel()
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/IVAlgoHTTP-test/algos/AnystyleImg2Img/newfilter.py", line 725, in __init__
    self.model.pipline.pipe.load_lora_weights(load_file(os.path.join(self.model_root, self.config['SDXL'][self.default_style]['lora_repo'][0]), device=self.device), adapter_name="mix_lora")
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/venv310/lib/python3.10/site-packages/diffusers/loaders/lora.py", line 1255, in load_lora_weights
    self.load_lora_into_text_encoder(
File "/home/iv/Algo_new/LouHaijie/IVAlgoHTTP/venv310/lib/python3.10/site-packages/diffusers/loaders/lora.py", line 555, in load_lora_into_text_encoder
    rank[rank_key] = text_encoder_lora_state_dict[rank_key].shape[1]
KeyError: 'text_model.encoder.layers.0.self_attn.out_proj.lora_B.weight'

三、如何在diffusers0.31.0版本使用flux-redux

1、安装diffuser0.31.0版本，基础模型的pipeline的类和方法都从该库中调用；

2、从github上安装diffusers0.32.0-release版本，flux-redux的类的方法都从该库中调用；

3、修改diffusers0.32.0-release库：

diffusers/pipelines/pipeline_utils.py：from diffusers import 修改为from [当前库名] import；

4、修改diffusers0.31.0库：diffusers/models/attention_processor.py:

hidden_states = F.scaled_dot_product_attention(query, key, value, dropout_p=0.0, is_causal=False)
修改为：
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False)

四、diffusers版本为0.31.0时，使用optimum量化，输出图片为噪声

修改optimum/quanto/tensor/weights/qbytes.py.

and torch.cuda.get_device_capability(data.device)[0] >= 8
为
and torch.cuda.get_device_capability(data.device)[0] >= 20

修改后：

if (
        qtype == qtypes["qfloat8_e4m3fn"]
        and activation_qtype is None
        and scale.dtype in [torch.float16, torch.bfloat16]
        and len(size) == 2
        and (data.device.type == "cuda" and torch.version.cuda)
        and axis == 0
        and torch.cuda.get_device_capability(data.device)[0] >= 8
    ):
        out_features, in_features = size
        if (
            in_features >= 64
            and out_features >= 64
            and (
                (in_features % 64 == 0 and out_features % 128 == 0)
                or (in_features % 128 == 0 and out_features % 64 == 0)
            )
        ):
            return MarlinF8QBytesTensor(qtype, axis, size, stride, data, scale, requires_grad)

五、diffusers要同时设两个lora权重时，有一个不生效果

可能是设权重的方式错了：

错误方式：

self.pipe.set_adapters([LoraA], adapter_weights=[A])

self.pipe.set_adapters([LoraB], adapter_weights=[B])

正确方式：

self.pipe.set_adapters([LoraA, LoraB], adapter_weights=[A, B])