使用Langfuse实现jxnl/instructor项目的可观测性与追踪-优快云博客

使用Langfuse实现jxnl/instructor项目的可观测性与追踪

instructor structured outputs for llms 项目地址: https://gitcode.com/gh_mirrors/in/instructor

什么是Langfuse？

Langfuse是一个开源的LLM工程平台，专为AI应用开发团队设计，提供API调用追踪、性能监控和问题调试等功能。在jxnl/instructor项目中，我们可以利用Langfuse来追踪和监控模型调用过程，这对于理解模型行为、优化性能和调试问题非常有帮助。

环境准备

在开始之前，请确保已经完成以下准备工作：

注册Langfuse账号并获取API密钥
安装必要的Python包

pip install langfuse instructor

基础集成方法

jxnl/instructor与Langfuse的集成非常简单，主要通过包装OpenAI客户端实现。下面分别介绍同步和异步两种集成方式。

同步客户端集成

import instructor
from langfuse.openai import openai
from pydantic import BaseModel
import os

# 配置环境变量
os.environ["LANGFUSE_PUBLIC_KEY"] = "你的公钥"
os.environ["LANGFUSE_SECRET_KEY"] = "你的私钥"
os.environ["LANGFUSE_HOST"] = "Langfuse服务地址"
os.environ["OPENAI_API_KEY"] = "OpenAI API密钥"

# 包装同步客户端
client = instructor.from_openai(openai.OpenAI())

class WeatherDetail(BaseModel):
    city: str
    temperature: int

# 执行模型调用
weather_info = client.chat.completions.create(
    model="gpt-4",
    response_model=WeatherDetail,
    messages=[
        {"role": "user", "content": "巴黎的气温是18摄氏度。"},
    ],
)

print(weather_info.model_dump_json(indent=2))

执行后，你可以在Langfuse仪表板中查看完整的调用追踪记录。

异步客户端集成

import instructor
from langfuse.openai import openai
from pydantic import BaseModel
import os
import asyncio

# 配置环境变量(同上)

# 包装异步客户端
client = instructor.from_openai(openai.AsyncOpenAI())

class WeatherDetail(BaseModel):
    city: str
    temperature: int

async def main():
    weather_info = await client.chat.completions.create(
        model="gpt-4",
        response_model=WeatherDetail,
        messages=[
            {"role": "user", "content": "巴黎的气温是18摄氏度。"},
        ],
    )
    print(weather_info.model_dump_json(indent=2))

asyncio.run(main())

实战案例：客户反馈分类与评分

下面我们通过一个更复杂的案例，展示如何结合jxnl/instructor和Langfuse实现客户反馈的自动分类和相关性评分。

1. 定义数据模型

首先，我们定义反馈类型枚举和分类模型：

from enum import Enum

class FeedbackType(Enum):
    PRAISE = "表扬"
    SUGGESTION = "建议"
    BUG = "缺陷"
    QUESTION = "问题"

class FeedbackClassification(BaseModel):
    feedback_text: str = Field(..., description="反馈内容")
    classification: list[FeedbackType] = Field(
        description="预测的反馈分类"
    )
    relevance_score: float = Field(
        default=0.0,
        description="反馈与业务的相关性评分，0.0到1.0之间",
        ge=0.0,
        le=1.0
    )

2. 实现分类函数

使用@observe装饰器自动记录追踪信息：

from langfuse.decorators import observe

@observe()
async def classify_feedback(feedback: str):
    async with sem:  # 简单的速率限制
        response = await client.chat.completions.create(
            model="gpt-4",
            response_model=FeedbackClassification,
            max_retries=2,
            messages=[
                {"role": "user", "content": f"分类并评分此反馈: {feedback}"},
            ],
        )
        return feedback, response

3. 评分功能实现

def score_relevance(trace_id: str, observation_id: str, score: float):
    langfuse.score(
        trace_id=trace_id,
        observation_id=observation_id,
        name="feedback-relevance",
        value=score,
        comment="用户反馈相关性评分"
    )

4. 主处理流程

@observe()
async def main(feedbacks: list[str]):
    tasks = [classify_feedback(feedback) for feedback in feedbacks]
    results = []
    
    for task in asyncio.as_completed(tasks):
        feedback, classification = await task
        result = {
            "feedback": feedback,
            "classification": [c.value for c in classification.classification],
            "relevance_score": classification.relevance_score,
        }
        results.append(result)
    
    return results

5. 执行示例

feedback_messages = [
    "你们网站的聊天机器人不能用。",
    "你们的客户服务太棒了！",
    "能否在应用中添加更多功能？",
    "我想咨询最近的订单情况。",
]

results = asyncio.run(main(feedback_messages))
for result in results:
    print(f"反馈: {result['feedback']}")
    print(f"分类: {result['classification']}")
    print(f"相关性评分: {result['relevance_score']}")