Milvus向量数据库学习—— Build RAG with Mivus

嗨还阔以

已于 2024-11-13 16:20:58 修改

阅读量550

点赞数 5

文章标签： milvus 全文检索 python 自然语言处理语言模型数据库

于 2024-11-12 11:06:03 首次发布

本文链接：https://blog.youkuaiyun.com/xs86467/article/details/143705900

版权

Milvus向量数据库学习—— Build RAG with Mivus

Milvus 是一个开源的向量数据库，用于存储、管理和检索高维向量数据。它特别适合需要快速近似最近邻搜索（ANN，Approximate Nearest Neighbor）的应用场景，如推荐系统、计算机视觉、自然语言处理等。Milvus 提供了高性能、可扩展的解决方案，可以方便地集成到现代 AI 应用中。
官方文档

1.Prepare the data

from glob import glob

text_lines = []

for file_path in glob("few shots.txt", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()

    text_lines += file_text.split("# ")
# 在这里我使用了 ‘# ’ 对文本块进行了分割
# 后续检索出的文本块会是被‘# ’分割的文本
print(type(text_lines))
print(text_lines)

2.Prepare the Embedding Model

from openai import OpenAI
openai_client = OpenAI(
    api_key = "Your api key here.....",
    base_url = "URL"
)
def emb_text(text):
    return (
        openai_client.embeddings.create(input=text, model="text-embedding-v3")
        .data[0]
        .embedding
    )

test_embedding = emb_text("This is a test")
embedding_dim = len(test_embedding)
print(embedding_dim)
print(test_embedding[:10])

3.Load data into Milvus

# 创建Collections
from pymilvus import MilvusClient

milvus_client = MilvusClient(uri="./milvus_demo.db") # 会存储在当前路径下

collection_name = "my_rag_collection" # 创建集合的名称

milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim, # 嵌入向量的维度
    metric_type="IP",  # 创建集合时应该定好采用什么样的metric_type
    consistency_level="Strong",  # Strong consistency level
)

consistency_level="Strong" 确保用户可以读取到最新版本的数据，设置集合的一致性级别为"强一致性"。Milvus支持四种一致性级别：Strong（强一致性）、Bounded（有界一致性）、Session（会话一致性）和Eventually（最终一致性）

About collection

Collection在Milvus中相当于关系型数据库中的Table
一个数据库中可以拥有多个Collection
不同维度的向量需存储在不同的Collection中
需要使用不同相似度计算方式的向量也需要存储在不同的Collection

# 插入数据
from tqdm import tqdm
data = []

for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": emb_text(line), "text": line})
milvus_client.insert(collection_name=collection_name, data=data)

{'insert_count': 72,
 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71],
 'cost': 0}

milvus_client.insert() 方法是增量插入，不会覆盖Collection中的其它内容。

# 插入单条记录
res = client.insert(
    collection_name="test_collection",
    data={
        'id': 0,
        'vector': [
            0.6186516144460161,
            0.5927442462488592,
            0.848608119657156,
            0.9287046808231654,
            -0.42215796530168403
        ]
    }
)

可选参数：

collection_name：String 类型
timeout :操作超时时间

4. Build RAG

Retrieve data for a query

question = "your question here...."

search_res = milvus_client.search(
    collection_name=collection_name,
    data=[
        emb_text(question)
    ],  # Use the `emb_text` function to convert the question to an embedding vector
    limit=3,  # Return top 3 results
    search_params={"metric_type": "IP", "params": {}},  # Inner product distance   # COSINE
    output_fields=["text"],  # Return the text field
)

这里使用的mertic_type 要与创建collection时所确定的一样。

[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.7883545756340027
    ],
    [
        "How does Milvus handle vector data types and precision?\n\nMilvus supports Binary, Float32, Float16, and BFloat16 vector types.\n\n- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.\n- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.\n- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.\n\n###",
        0.6757288575172424
    ],
    [
        "How much does Milvus cost?\n\nMilvus is a 100% free open-source project.\n\nPlease adhere to [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0) when using Milvus for production or distribution purposes.\n\nZilliz, the company behind Milvus, also offers a fully managed cloud version of the platform for those that don't want to build and maintain their own distributed instance. [Zilliz Cloud](https://zilliz.com/cloud) automatically maintains data reliability and allows users to pay only for what they use.\n\n###",
        0.6421123147010803
    ]
]