多模态大模型 deepseek-ai/Janus-Pro-1B

Janus-Pro-1B 模型介绍

Janus-Pro是一种新颖的自回归框架，它统一了多模态的理解和生成。该框架通过将视觉编码分解为独立的路径，解决了以往方法的局限性，同时仍然使用单一的统一Transformer架构进行处理。这种解耦不仅缓解了视觉编码器在理解和生成中的角色冲突，还增强了框架的灵活性。Janus-Pro超越了以往的统一模型，并在性能上与特定任务的模型相当甚至超过它们。Janus-Pro的简洁性、高灵活性和有效性使其成为下一代统一多模态模型的有力候选。

• github:

  Github: https://github.com/deepseek-ai/Janus

• 性能

  

• 生成效果示例

  

• Model Summary

  Janus-Pro是基于DeepSeek-LLM-1.5b-base构建的。
  在多模态理解方面，它使用SigLIP-L作为视觉编码器，支持384×384像素的图像输入。在图像生成方面，Janus-Pro使用了这里的标记器，下采样率为16。

• 发布时间

  2025年1月28日

下载

model_id: deepseek-ai/Janus-Pro-1B
下载地址：[https://hf-mirror.com/deepseek-ai/Janus-Pro-1B](https://hf-mirror.com/deepseek-ai/Janus-Pro-1B)   不需要翻墙

运行环境安装

git clone https://github.com/deepseek-ai/Janus
cd Janus
pip install -e .  -i https://pypi.mirrors.ustc.edu.cn/simple

模型推理示例

• Multimodal Understanding

  
  import torch
  from transformers import AutoModelForCausalLM
  from janus.models import MultiModalityCausalLM, VLChatProcessor
  from janus.utils.io import load_pil_images
  
  # specify the path to the model
  model_path = "deepseek-ai/Janus-Pro-1B"
  vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
  tokenizer = vl_chat_processor.tokenizer
  
  vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
      model_path, trust_remote_code=True
  )
  vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()
  
  conversation = [
      {
          "role": "<|User|>",
          "content": f"<image_placeholder>\n{question}",
          "images": [image],
      },
      {"role": "<|Assistant|>", "content": ""},
  ]
  
  # load images and prepare for inputs
  pil_images = load_pil_images(conversation)
  prepare_inputs = vl_chat_processor(
      conversations=conversation, images=pil_images, force_batchify=True
  ).to(vl_gpt.device)
  
  # # run image encoder to get the image embeddings
  inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)
  
  # # run the model to get the response
  outputs = vl_gpt.language_model.generate(
      inputs_embeds=inputs_embeds,
      attention_mask=prepare_inputs.attention_mask,
      pad_token_id=tokenizer.eos_token_id,
      bos_token_id=tokenizer.bos_token_id,
      eos_token_id=tokenizer.eos_token_id,
      max_new_tokens=512,
      do_sample=False,
      use_cache=True,
  )
  
  answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
  print(f"{prepare_inputs['sft_format'][0]}", answer)
  
  

• Text-to-Image Generation

  import os
  import PIL.Image
  import torch
  import numpy as np
  from transformers import AutoModelForCausalLM
  from janus.models import MultiModalityCausalLM, VLChatProcessor
  
  
  # specify the path to the model
  model_path = "deepseek-ai/Janus-Pro-1B"
  vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
  tokenizer = vl_chat_processor.tokenizer
  
  vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
      model_path, trust_remote_code=True
  )
  vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()
  
  conversation = [
      {
          "role": "<|User|>",
          "content": "A stunning princess from kabul in red, white traditional clothing, blue eyes, brown hair",
      },
      {"role": "<|Assistant|>", "content": ""},
  ]
  
  sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(
      conversations=conversation,
      sft_format=vl_chat_processor.sft_format,
      system_prompt="",
  )
  prompt = sft_format + vl_chat_processor.image_start_tag
  
  
  @torch.inference_mode()
  def generate(
      mmgpt: MultiModalityCausalLM,
      vl_chat_processor: VLChatProcessor,
      prompt: str,
      temperature: float = 1,
      parallel_size: int = 16,
      cfg_weight: float = 5,
      image_token_num_per_image: int = 576,
      img_size: int = 384,
      patch_size: int = 16,
  ):
      input_ids = vl_chat_processor.tokenizer.encode(prompt)
      input_ids = torch.LongTensor(input_ids)
  
      tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda()
      for i in range(parallel_size*2):
          tokens[i, :] = input_ids
          if i % 2 != 0:
              tokens[i, 1:-1] = vl_chat_processor.pad_id
  
      inputs_embeds = mmgpt.language_model.get_input_embeddings()(tokens)
  
      generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()
  
      for i in range(image_token_num_per_image):
          outputs = mmgpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)
          hidden_states = outputs.last_hidden_state
          
          logits = mmgpt.gen_head(hidden_states[:, -1, :])
          logit_cond = logits[0::2, :]
          logit_uncond = logits[1::2, :]
          
          logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)
          probs = torch.softmax(logits / temperature, dim=-1)
  
          next_token = torch.multinomial(probs, num_samples=1)
          generated_tokens[:, i] = next_token.squeeze(dim=-1)
  
          next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)
          img_embeds = mmgpt.prepare_gen_img_embeds(next_token)
          inputs_embeds = img_embeds.unsqueeze(dim=1)
  
  
      dec = mmgpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size])
      dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)
  
      dec = np.clip((dec + 1) / 2 * 255, 0, 255)
  
      visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)
      visual_img[:, :, :] = dec
  
      os.makedirs('generated_samples', exist_ok=True)
      for i in range(parallel_size):
          save_path = os.path.join('generated_samples', "img_{}.jpg".format(i))
          PIL.Image.fromarray(visual_img[i]).save(save_path)
  
  
  generate(
      vl_gpt,
      vl_chat_processor,
      prompt,
  )
  

• Gradio Demo

  pip install -e .[gradio]
  
  python demo/app_januspro.py
  

开源协议

This code repository is licensed under the MIT License. The use of Janus-Pro models is subject to DeepSeek Model License.

参考

• https://hf-mirror.com/deepseek-ai/Janus-Pro-1B
• https://github.com/deepseek-ai/Janus