**用AI轻松为图像生成描述：深入探索Image Captioning**

本文链接：https://blog.youkuaiyun.com/tt_jishu/article/details/143411191

引言

随着视觉内容在数字世界中的丰富，如何高效地理解和描述图像变得越来越重要。图像字幕生成（Image Captioning）是计算机视觉领域中的一个重要课题，它可以帮助机器理解图像内容，并将其转化为自然语言描述。本文将介绍如何利用Salesforce的BLIP模型和ImageCaptionLoader来生成可查询的图像描述索引，为进一步的问答任务做好准备。

主要内容

1. 安装必要的库

首先，我们需要安装一些库来支持我们的工作。

%pip install -qU transformers langchain_openai langchain_chroma

2. 准备图像数据

我们将从Wikimedia中获取一些图像，并使用ImageCaptionLoader进行处理。

import getpass
import os

os.environ["OPENAI_API_KEY"] = getpass.getpass()

from langchain_community.document_loaders import ImageCaptionLoader

list_image_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Ara_ararauna_Luc_Viatour.jpg/1554px-Ara_ararauna_Luc_Viatour.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/0/0c/1928_Model_A_Ford.jpg/640px-1928_Model_A_Ford.jpg",
]

3. 创建图像描述加载器

使用ImageCaptionLoader来生成图像描述。

loader = ImageCaptionLoader(images=list_image_urls)
list_docs = loader.load()
print(list_docs)

输出结果展示了每张图像的描述，便于后续的处理。

4. 创建索引

为了能查询这些描述，我们需要将它们转换成可检索的索引。

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(list_docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())

retriever = vectorstore.as_retriever(k=2)

代码示例

接下来，我们将展示如何结合生成的索引进行问答。

from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o", temperature=0)

system_prompt = (
    "You are an assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer "
    "the question. If you don't know the answer, say that you "
    "don't know. Use three sentences maximum and keep the "
    "answer concise."
    "\n\n"
    "{context}"
)

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        ("human", "{input}"),
    ]
)

question_answer_chain = create_stuff_documents_chain(model, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

response = rag_chain.invoke({"input": "What kind of images are there?"})

print(response["answer"])