6-langchang多模态输入和自定义输出

多模态数据输入

这里我们演示如何将多模态输入直接传递给模型。我们目前期望所有输入都以与OpenAI 期望的格式相同的格式传递。对于支持多模态输入的其他模型提供者，我们在类中添加了逻辑以转换为预期格式。

在这个例子中，我们将要求模型描述一幅图像。

url

import base64
import httpx
from langchain_core.messages import HumanMessage

from langchain_openai.chat_models.base import BaseChatOpenAI

img_url = "https://nwzimg.wezhan.cn/contents/sitefiles2054/10270786/images/28839137.jpg"
llm = BaseChatOpenAI(
    # 需要换一个有视觉能力的模型
    model='qwen-omni-turbo',
    openai_api_base='https://dashscope.aliyuncs.com/compatible-mode/v1',
    max_tokens=1024
)

message = HumanMessage(content=[
    {
   "type": "text", "text": "用中文描述图片的天气"},
    {
   "type": "image_url", "image_url": {
   "url": img_url}},
])


chunks = []
response = llm.stream([message])
for chunk in response:
    chunks.append(chunk)
    print(chunk.content, end='|', flush=True)

|这张|图片|显示|的是|晴|朗|的|天气|。|天空|湛|蓝|，|阳光|明媚|，|太阳|高|挂在|空中|，|发出|耀眼|的|光芒|。|几|朵|白云|悠闲|地|飘|浮|在|蓝天|上|，|整个|画面|给人一种|清新|、|明亮|的感觉|。|这样的|天气|通常|适合|户外|活动|。|||
d

base64

import base64
import httpx
from langchain_core.messages import HumanMessage

from langchain_openai.chat_models.base import BaseChatOpenAI

img_url = "https://nwzimg.wezhan.cn/contents/sitefiles2054/10270786/images/28839137.jpg"
img_data = base64.b64encode(httpx.get(img_url).content).decode('utf-8')
llm = BaseChatOpenAI(
    # 需要换一个有视觉能力的模型
    model='qwen-omni-turbo',
    openai_api_base='https://dashscope.aliyuncs.com/compatible-mode/v1',
    max_tokens=1024,
# 设置输出数据的模态，当前支持["text"]
    modalities=["text"],
    # stream 必须设置为 True，否则会报错
    stream=True,
    stream_options={
   "include_usage": True},
)

message = HumanMessage(content=[
    {
   "type": "text", "text": "用中文描述图片的天气"},
    {
   "type": "image_url", "image_url": {
   "url": f"data:image/jpg;base64,{
     img_data}"}},
])

chunks = []
response = llm.stream([message])
for chunk in response:
    chunks.append(chunk)
    print(chunk.content, end='|', flush=True)

url list

import base64
import httpx
from langchain_core.messages import HumanMessage

from langchain_openai.chat_models.base import BaseChatOpenAI

llm = BaseChatOpenAI(
    # 需要换一个有视觉能力的模型
    model='qwen-omni-turbo',
    openai_api_base='https://dashscope.aliyuncs.com/compatible-mode/v1',
    max_tokens=1024,

# 设置输出数据的模态，当前支持["text"]
    modalities=["text"],
    # stream 必须设置为 True，否则会报错
    stream=True,
    stream_options={
   "include_usage": True},
)


img_url1 = "https://nwzimg.wezhan.cn/contents/sitefiles2054/10270786/images/28839137.jpg"
img_url2 = "https://nwzimg.wezhan.cn/contents/sitefiles2054/10270786/images/28839137.jpg"
message = HumanMessage(content=[
    {
   "type": "text", "text": "这两张图片是一样的吗？"},
    {
   "type": "image_url", "image_url": {
   "url": img_url1}},
    {
   "type": "image_url", "image_url": {
   "url": img_url2}},
])


chunks = [