Cohere reranker 一致的排序器

这本notebook展示了如何在检索器中使用 Cohere 的重排端点。这是在 ContextualCompressionRetriever 的想法基础上构建的。

%pip install --upgrade --quiet  cohere

%pip install --upgrade --quiet  faiss

# OR  (depending on Python version)

%pip install --upgrade --quiet  faiss-cpu

# get a new token: https://dashboard.cohere.ai/

import getpass
import os

os.environ["COHERE_API_KEY"] = getpass.getpass("Cohere API Key:")

# get a new token: https://dashboard.cohere.ai/

import getpass
import os

os.environ["COHERE_API_KEY"] = getpass.getpass("Cohere API Key:")

建立基础矢量存储检索器

让我们首先初始化一个简单的向量存储检索器，并存储 2023 年（分块）。我们可以设置检索器以检索大量（20）的文档。

from langchain_community.document_loaders import TextLoader
from langchain_community.embeddings import CohereEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_text_splitters import RecursiveCharacterTextSplitter

documents = TextLoader("../../how_to/state_of_the_union.txt").load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)
texts = text_splitter.split_documents(documents)
retriever = FAISS.from_documents(texts, CohereEmbeddings()).as_retriever(
    search_kwargs={"k": 20}
)

query = "What did the president say about Ketanji Brown Jackson"
docs = retriever.invoke(query)
pretty_print_docs(docs)

API 参考：TextLoader | CohereEmbeddings | FAISS | RecursiveCharacterTextSplitter

Document 1:

One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. 

And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
----------------------------------------------------------------------------------------------------
Document 2:

We cannot let this happen. 

Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. 

Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.
----------------------------------------------------------------------------------------------------
Document 3:

As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential. 

While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice.
----------------------------------------------------------------------------------------------------

使用 CohereRerank 进行重新排名

现在让我们用一个 ContextualCompressionRetriever 包装我们的基础检索器。我们会添加一个 CohereRerank ，使用 Cohere 重新排名端点来重新排列返回的结果。

from langchain.retrievers.contextual_compression import ContextualCompressionRetriever
from langchain_cohere import CohereRerank
from langchain_community.llms import Cohere

llm = Cohere(temperature=0)
compressor = CohereRerank()
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor, base_retriever=retriever
)

compressed_docs = compression_retriever.invoke(
    "What did the president say about Ketanji Jackson Brown"
)
pretty_print_docs(compressed_docs)

API 参考: ContextualCompressionRetriever | CohereRerank | Cohere

您当然可以在问答管道中使用这个检索器

from langchain.chains import RetrievalQA

API 参考：RetrievalQA

chain = RetrievalQA.from_chain_type(
    llm=Cohere(temperature=0), retriever=compression_retriever
)

chain({"query": query})

{'query': 'What did the president say about Ketanji Brown Jackson',
 'result': " The president speaks highly of Ketanji Brown Jackson, stating that she is one of the nation's top legal minds, and will continue the legacy of excellence of Justice Breyer. The president also mentions that he worked with her family and that she comes from a family of public school educators and police officers. Since her nomination, she has received support from various groups, including the Fraternal Order of Police and judges from both major political parties. \n\nWould you like me to extract another sentence from the provided text? "}

Edit this page