从图像生成可查询标题：使用ImageCaptionLoader和Chroma的最佳实践-优快云博客

本文链接：https://blog.youkuaiyun.com/tt_jishu/article/details/143681165

# 引言

在处理大量图像数据时，给每张图像自动生成描述性标题是一个有趣且重要的任务。这样做能帮助我们快速索引和检索图像，特别是在使用自然语言处理模型进行查询时。本文将介绍如何使用`ImageCaptionLoader`结合`Chroma`和`OpenAI`的工具快速生成和查询图像标题。

# 主要内容

## 1. 设置开发环境

首先，需要安装必要的Python库：

```bash
%pip install -qU transformers langchain_openai langchain_chroma

2. 准备图像数据

我们从Wikimedia准备了一组图像链接，这些图像将被用于生成描述性标题。

from langchain_community.document_loaders import ImageCaptionLoader

list_image_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Ara_ararauna_Luc_Viatour.jpg/1554px-Ara_ararauna_Luc_Viatour.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/0/0c/1928_Model_A_Ford.jpg/640px-1928_Model_A_Ford.jpg",
]

3. 生成图像标题

使用ImageCaptionLoader加载并生成图像描述。ImageCaptionLoader利用了预训练的Salesforce BLIP模型来生成图像的标题。

loader = ImageCaptionLoader(images=list_image_urls)
list_docs = loader.load()
print(list_docs)

4. 创建图像索引

接下来，我们使用Chroma库和OpenAIEmbeddings创建图像描述的可查询向量索引。

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(list_docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())
retriever = vectorstore.as_retriever(k=2)

5. 查询生成的图像标题

为了实现从索引中查询图像标题，我们使用OpenAI的模型和自定义提示模板创建问答链。

from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o", temperature=0)
system_prompt = (
    "You are an assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer "
    "the question. If you don't know the answer, say that you "
    "don't know. Use three sentences maximum and keep the answer concise."
    "\n\n"
    "{context}"
)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        ("human", "{input}"),
    ]
)
question_answer_chain = create_stuff_documents_chain(model, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

response = rag_chain.invoke({"input": "What kind of images are there?"})
print(response["answer"])