基于Google Cloud Platform构建实时机器学习推理系统-优快云博客

本文链接：https://blog.youkuaiyun.com/gitblog_00383/article/details/148441318

基于Google Cloud Platform构建实时机器学习推理系统

training-data-analyst Labs and demos for courses for GCP Training (http://cloud.google.com/training). 项目地址: https://gitcode.com/gh_mirrors/tr/training-data-analyst

本文将详细介绍如何在Google Cloud Platform上构建一个能够处理实时数据的机器学习推理系统。我们将以出租车费用预测模型为例，展示如何整合实时交通数据来提升模型预测的准确性。

实时数据处理架构概述

构建实时机器学习推理系统需要以下几个关键组件：

数据生成层：由物联网设备(如出租车)实时生成并发送数据
消息总线：接收并临时存储实时数据(使用Cloud Pub/Sub)
流处理服务：订阅消息总线，对数据进行窗口化和转换处理(使用Cloud Dataflow)
持久化存储：保存处理后的数据(使用BigQuery)

这种架构允许我们持续不断地处理实时数据流，并在模型推理时获取最新的实时信息。

环境准备与初始化

首先我们需要设置项目环境变量并导入必要的Python库：

import numpy as np
import os
import shutil
import tensorflow as tf

from google.cloud import aiplatform
from google.cloud import bigquery
from google.protobuf import json_format
from google.protobuf.struct_pb2 import Value
from matplotlib import pyplot as plt
from tensorflow import keras
from tensorflow.keras.callbacks import TensorBoard
from tensorflow.keras.layers import Dense, DenseFeatures
from tensorflow.keras.models import Sequential

# 设置项目环境变量
PROJECT = 'your-project-id'  # 替换为你的项目ID
BUCKET = 'your-bucket-name'  # 替换为你的存储桶名称
REGION = 'us-central1'       # 替换为你的区域

os.environ['PROJECT'] = PROJECT
os.environ['BUCKET'] = BUCKET
os.environ['REGION'] = REGION

模型重新训练

为了利用实时数据，我们需要在原有模型基础上增加trips_last_5min特征：

# 模型构建示例代码
def build_model():
    model = Sequential([
        DenseFeatures(feature_columns),
        Dense(128, activation='relu'),
        Dense(128, activation='relu'),
        Dense(1)
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

这个新增特征将反映最近5分钟的交通流量情况，帮助模型做出更准确的预测。

实时数据模拟与处理

1. 模拟实时数据

由于没有真实的实时出租车数据，我们可以使用Python脚本模拟：

# 模拟数据发布到Pub/Sub
def publish_messages(project_id, topic_name):
    publisher = pubsub_v1.PublisherClient()
    topic_path = publisher.topic_path(project_id, topic_name)
    
    while True:
        data = generate_taxi_data()  # 生成模拟数据
        future = publisher.publish(topic_path, data=data.encode('utf-8'))
        time.sleep(random.uniform(0.5, 2.0))  # 随机间隔模拟真实流量

2. 创建BigQuery表存储处理后的数据

# 创建BigQuery表和数据集
def create_bq_table():
    client = bigquery.Client()
    dataset_ref = client.dataset('taxifare')
    
    try:
        dataset = bigquery.Dataset(dataset_ref)
        client.create_dataset(dataset)
        print("Dataset created")
    except Exception as e:
        print("Dataset already exists")
    
    schema = [
        bigquery.SchemaField("trips_last_5min", "INTEGER", mode="REQUIRED"),
        bigquery.SchemaField("time", "TIMESTAMP", mode="REQUIRED")
    ]
    
    table_ref = dataset_ref.table("traffic_realtime")
    table = bigquery.Table(table_ref, schema=schema)
    
    try:
        client.create_table(table)
        print("Table created")
    except Exception as e:
        print("Table already exists")

3. 构建Dataflow流处理管道

# 流处理管道核心逻辑
def run_pipeline(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument('--input_topic', required=True)
    known_args, pipeline_args = parser.parse_known_args(argv)
    
    pipeline_options = PipelineOptions(pipeline_args)
    
    with beam.Pipeline(options=pipeline_options) as p:
        (p
         | 'Read from PubSub' >> beam.io.ReadFromPubSub(topic=known_args.input_topic)
         | 'Window into 5min' >> beam.WindowInto(
             beam.window.SlidingWindows(300, 15))  # 5分钟窗口，每15秒滑动一次
         | 'Count messages' >> beam.combiners.Count.Globally()
         | 'Format for BQ' >> beam.Map(lambda count: {
             'trips_last_5min': count,
             'time': datetime.now().isoformat()
         })
         | 'Write to BQ' >> beam.io.WriteToBigQuery(
             'taxifare.traffic_realtime',
             schema='trips_last_5min:INTEGER,time:TIMESTAMP',
             create_disposition=beam.io.BigQueryDisposition.CREATE_NEVER,
             write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))

实时预测实现

1. 获取最新交通数据

def get_recent_traffic():
    client = bigquery.Client()
    query = """
        SELECT trips_last_5min 
        FROM `taxifare.traffic_realtime` 
        ORDER BY time DESC 
        LIMIT 1
    """
    result = client.query(query).to_dataframe()
    return result['trips_last_5min'][0]

2. 整合实时数据进行预测

def predict_with_realtime_data(instance):
    # 添加实时交通数据
    instance['trips_last_5min'] = get_recent_traffic()
    
    # 准备预测请求
    client = aiplatform.gapic.PredictionServiceClient(
        client_options={"api_endpoint": f"{REGION}-aiplatform.googleapis.com"}
    )
    
    # 转换实例格式
    instance_proto = json_format.ParseDict(instance, Value())
    
    # 发送预测请求
    response = client.predict(
        endpoint=f"projects/{PROJECT}/locations/{REGION}/endpoints/{ENDPOINT_ID}",
        instances=[instance_proto]
    )
    
    # 解析并返回结果
    return response.predictions[0]['output_0'][0]