Quivr服务发现：Consul/Eureka服务注册和发现-优快云博客

Quivr服务发现：Consul/Eureka服务注册和发现

【免费下载链接】quivr Quivr，作为你的第二大脑，充分利用生成式人工智能（Generative AI）的力量成为你的个人智能助手！项目地址: https://gitcode.com/GitHub_Trending/qui/quivr

引言：为什么AI助手需要服务发现？

在构建现代化的AI助手系统时，微服务架构已成为主流选择。Quivr作为你的第二大脑，由多个协同工作的服务组成：前端界面、后端API、任务处理Worker、定时任务调度器等。当这些服务数量增多、部署环境复杂时，如何让它们自动发现彼此、实现负载均衡和高可用性，就成为了一个关键挑战。

传统的硬编码服务地址方式已经无法满足动态扩展的需求。本文将深入探讨如何在Quivr中集成Consul和Eureka服务发现机制，构建真正弹性的AI助手架构。

Quivr现有架构分析

当前服务组成

从Quivr的Docker Compose配置可以看出，系统包含以下核心服务：

服务名称	端口	功能描述	健康检查端点
frontend	3000	前端界面服务	-
backend-core	5050	核心API服务	`/healthz`
worker	-	异步任务处理	-
beat	-	定时任务调度	-
flower	5555	Celery监控界面	-
redis	6379	缓存和消息队列	-

现有健康检查机制

Quivr已经在多个模块中实现了基础的健康检查：

# backend/api/quivr_api/modules/misc/controller/misc_routes.py
@misc_router.get("/healthz", tags=["Health"])
async def healthz():
    return {"status": "ok"}

这种简单的健康检查为服务发现集成提供了良好基础。

Consul服务发现集成方案

Consul简介

Consul是HashiCorp开发的服务网格解决方案，提供服务发现、健康检查、键值存储和多数据中心支持。其基于Gossip协议的分布式架构非常适合微服务环境。

集成步骤

1. 添加Consul到Docker Compose

services:
  consul-server:
    image: consul:1.15
    container_name: consul-server
    command: agent -server -bootstrap-expect=1 -ui -client=0.0.0.0
    ports:
      - "8500:8500"
    volumes:
      - consul-data:/consul/data
    networks:
      - quivr-network

  consul-agent:
    image: consul:1.15
    container_name: consul-agent
    command: agent -retry-join=consul-server -client=0.0.0.0
    depends_on:
      - consul-server
    networks:
      - quivr-network

volumes:
  consul-data:

2. 创建Consul服务注册客户端

# backend/api/quivr_api/services/consul_client.py
import consul
import socket
from fastapi import FastAPI
from contextlib import asynccontextmanager

class ConsulClient:
    def __init__(self, host='consul-server', port=8500):
        self.consul = consul.Consul(host=host, port=port)
        self.service_id = None
    
    def register_service(self, service_name, port, tags=None):
        """注册服务到Consul"""
        hostname = socket.gethostname()
        service_id = f"{service_name}-{hostname}"
        
        check = consul.Check.http(
            f"http://{hostname}:{port}/healthz",
            interval="10s",
            timeout="5s",
            deregister="1m"
        )
        
        self.consul.agent.service.register(
            service_name,
            service_id=service_id,
            address=hostname,
            port=port,
            tags=tags or [],
            check=check
        )
        self.service_id = service_id
        return service_id
    
    def deregister_service(self):
        """从Consul注销服务"""
        if self.service_id:
            self.consul.agent.service.deregister(self.service_id)
    
    def discover_service(self, service_name):
        """发现指定服务"""
        index, data = self.consul.health.service(service_name, passing=True)
        return [f"{item['Service']['Address']}:{item['Service']['Port']}" 
                for item in data]

consul_client = ConsulClient()

@asynccontextmanager
async def lifespan(app: FastAPI):
    # 启动时注册服务
    consul_client.register_service("quivr-backend", 5050, ["api", "backend"])
    yield
    # 关闭时注销服务
    consul_client.deregister_service()

3. 修改主应用集成生命周期管理

# backend/api/quivr_api/main.py
from quivr_api.services.consul_client import lifespan

app = FastAPI(lifespan=lifespan)

4. 服务发现客户端实现

# backend/core/quivr_core/utils/service_discovery.py
import requests
from typing import List
from consul import Consul

class ServiceDiscovery:
    def __init__(self, consul_host='consul-server', consul_port=8500):
        self.consul = Consul(host=consul_host, port=consonul_port)
    
    def get_service_url(self, service_name: str) -> str:
        """获取服务URL"""
        index, data = self.consul.health.service(service_name, passing=True)
        if not data:
            raise Exception(f"Service {service_name} not found")
        
        # 简单的负载均衡：随机选择健康实例
        import random
        service = random.choice(data)
        return f"http://{service['Service']['Address']}:{service['Service']['Port']}"
    
    def get_all_services(self) -> List[str]:
        """获取所有注册的服务"""
        return self.consul.agent.services().keys()

Eureka服务发现集成方案

Eureka简介

Eureka是Netflix开源的服务发现组件，是Spring Cloud生态系统的核心组成部分。采用客户端发现模式，服务实例自己注册到Eureka服务器。

集成步骤

1. 添加Eureka到Docker Compose

services:
  eureka-server:
    image: springcloud/eureka
    container_name: eureka-server
    ports:
      - "8761:8761"
    environment:
      - SPRING_PROFILES_ACTIVE=development
    networks:
      - quivr-network

2. Python Eureka客户端集成

# backend/api/quivr_api/services/eureka_client.py
import py_eureka_client.eureka_client as eureka_client
from fastapi import FastAPI
from contextlib import asynccontextmanager

class EurekaClient:
    def __init__(self, eureka_server="http://eureka-server:8761/eureka",
                 app_name="quivr-backend", instance_port=5050):
        self.eureka_server = eureka_server
        self.app_name = app_name
        self.instance_port = instance_port
    
    async def register_service(self):
        """注册服务到Eureka"""
        await eureka_client.init_async(
            eureka_server=self.eureka_server,
            app_name=self.app_name,
            instance_port=self.instance_port,
            instance_host="localhost",
            health_check_url=f"http://localhost:{self.instance_port}/healthz",
            status_page_url=f"http://localhost:{self.instance_port}/healthz"
        )
    
    async def deregister_service(self):
        """从Eureka注销服务"""
        await eureka_client.stop_async()

eureka_client_instance = EurekaClient()

@asynccontextmanager
async def lifespan(app: FastAPI):
    # 启动时注册服务
    await eureka_client_instance.register_service()
    yield
    # 关闭时注销服务
    await eureka_client_instance.deregister_service()

3. 服务发现客户端

# backend/core/quivr_core/utils/eureka_discovery.py
import aiohttp
import json
from typing import List, Dict

class EurekaDiscovery:
    def __init__(self, eureka_url="http://eureka-server:8761"):
        self.eureka_url = eureka_url
    
    async def get_service_instances(self, app_name: str) -> List[Dict]:
        """获取服务实例列表"""
        url = f"{self.eureka_url}/eureka/apps/{app_name.upper()}"
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                data = await response.text()
                apps = json.loads(data)
                return apps['application']['instance']
    
    async def get_service_url(self, app_name: str) -> str:
        """获取服务URL"""
        instances = await self.get_service_instances(app_name)
        if not instances:
            raise Exception(f"No instances found for {app_name}")
        
        # 选择第一个健康实例
        instance = instances[0]
        return f"http://{instance['ipAddr']}:{instance['port']['$']}"

对比分析：Consul vs Eureka

特性	Consul	Eureka	Quivr推荐
服务发现	✅ 支持	✅ 支持	✅ 两者都适用
健康检查	✅ 丰富类型	✅ 基础检查	🔶 Consul更灵活
配置管理	✅ 键值存储	❌ 不支持	✅ Consul优势
多数据中心	✅ 原生支持	❌ 不支持	✅ Consul优势
监控界面	✅ Web UI	✅ Web UI	✅ 两者都有
社区生态	✅ 活跃	✅ 稳定	✅ 两者都好
语言支持	✅ 多语言	⚠️ Java为主	🔶 Consul更通用

实战：在Quivr中实现动态服务调用

场景：跨服务文件处理

当用户上传文件时，Quivr需要将文件处理任务分发到可用的Worker节点：

# backend/api/quivr_api/modules/upload/controller/upload_routes.py
from quivr_core.utils.service_discovery import ServiceDiscovery

class EnhancedUploadService:
    def __init__(self):
        self.service_discovery = ServiceDiscovery()
    
    async def process_file(self, file, brain_id):
        # 动态发现可用的worker服务
        worker_url = self.service_discovery.get_service_url("quivr-worker")
        
        # 使用发现的URL调用服务
        async with aiohttp.ClientSession() as session:
            async with session.post(
                f"{worker_url}/process",
                json={"file": file, "brain_id": brain_id}
            ) as response:
                return await response.json()

配置管理集成

利用Consul的配置管理功能，实现动态配置更新：

# backend/core/quivr_core/config.py
from quivr_api.services.consul_client import consul_client

class DynamicConfig:
    def __init__(self):
        self.consul = consul_client.consul
    
    def get_config(self, key, default=None):
        """从Consul获取配置"""
        index, data = self.consul.kv.get(key)
        return data['Value'].decode() if data else default
    
    def watch_config(self, key, callback):
        """监听配置变化"""
        index = None
        while True:
            index, data = self.consul.kv.get(key, index=index)
            if data:
                callback(data['Value'].decode())

部署和运维考虑

生产环境配置

# docker-compose.prod.yml
services:
  consul-server:
    deploy:
      replicas: 3
      placement:
        constraints:
          - node.role == manager
    configs:
      - source: consul-config
        target: /consul/config/config.json

  quivr-backend:
    deploy:
      replicas: 3
    environment:
      - CONSUL_HOST=consul-server
      - CONSUL_PORT=8500

监控和告警

# backend/api/quivr_api/monitoring/service_health.py
import schedule
import time
from quivr_core.utils.service_discovery import ServiceDiscovery

def monitor_services():
    discovery = ServiceDiscovery()
    services = discovery.get_all_services()
    
    for service in services:
        instances = discovery.get_service_instances(service)
        if len(instances) < 2:  # 最少需要2个实例
            send_alert(f"Service {service} has only {len(instances)} instances")

# 定时执行监控
schedule.every(5).minutes.do(monitor_services)

性能优化建议

1. 客户端缓存

class CachedServiceDiscovery(ServiceDiscovery):
    def __init__(self, ttl=30):  # 30秒缓存
        super().__init__()
        self.cache = {}
        self.ttl = ttl
    
    def get_service_url(self, service_name):
        current_time = time.time()
        if (service_name in self.cache and 
            current_time - self.cache[service_name]['timestamp'] < self.ttl):
            return self.cache[service_name]['url']
        
        url = super().get_service_url(service_name)
        self.cache[service_name] = {'url': url, 'timestamp': current_time}
        return url

2. 连接池管理

import aiohttp
from aiohttp import ClientSession, TCPConnector

class ServiceClient:
    def __init__(self):
        self.connector = TCPConnector(limit=100, limit_per_host=20)
        self.session = ClientSession(connector=self.connector)
    
    async def call_service(self, service_name, endpoint, **kwargs):
        url = f"{self.discovery.get_service_url(service_name)}/{endpoint}"
        async with self.session.get(url, **kwargs) as response:
            return await response.json()

总结

通过集成Consul或Eureka服务发现机制，Quivr能够实现：

自动服务注册与发现：新服务实例自动加入集群
负载均衡：智能分配请求到健康实例
故障恢复：自动剔除不健康实例
动态配置：实时更新配置而不重启服务

创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考