目录
2.3 视觉模态：CNN/Vision Transformer特征提取
Algorithm 3: 视觉Transformer特征提取与注意力引导
脚本 2.1：文本模态Transformer特征提取与优化系统
脚本 2.3：视觉模态CNN/Vision Transformer特征提取系统
第一部分：原理详解
2.1 文本模态：Transformer特征提取与优化
基于自注意力机制的Transformer架构已成为文本情感特征提取的主流范式。该架构通过多头注意力机制（Multi-Head Attention）实现对长程上下文依赖的建模，克服了传统循环神经网络在长序列处理中的梯度消失问题。
2.1.1 双向编码器表征架构
BERT（Bidirectional Encoder Representations from Transformers）采用双向Transformer编码器，通过掩码语言模型（Masked Language Model, MLM）和下一句预测（Next Sentence Prediction, NSP）进行预训练。对于情感识别任务，给定输入序列 $x=[x_1, x_2, \dots, x_T]$，其隐藏状态表征通过以下计算获得：
$$H = \text{TransformerEncoder}(x) \in \mathbb{R}^{T \times d}$$
其中 $d$ 为隐藏层维度（Base版本 $d=768$，Large版本 $d=1024$）。在情感分类任务中，通常采用最后一层隐藏状态的分类令牌（[CLS] token）作为句级表征：
$$h_{\text{CLS}} = H[0,:] \in \mathbb{R}^d$$
RoBERTa（Robustly Optimized BERT Approach）通过移除NSP任务、采用动态掩码策略及更大规模的训练数据，进一步优化了上下文表征能力。EmoBERTa在此基础上针对情感识别任务进行了领域自适应预训练，在对话情感识别中表现出更强的判别力。
2.1.2 分层注意力优化机制
在多模态情感识别中，文本特征需与异构模态进行语义对齐。分层注意力策略（Hierarchical Attention Strategy）首先在词级别计算自注意力权重：
$$\alpha_{ij} = \frac{\exp(q_i^T k_j / \sqrt{d_k})}{\sum_{l=1}^T \exp(q_i^T k_l / \sqrt{d_k})}$$
其中 $q_i = W_Q h_i$，$k_j = W_K h_j$，$d_k$ 为键向量的维度。注意力输出为：
$$o_i = \sum_{j=1}^T \alpha_{ij} v_j$$
其次，在话语级别引入上下文注意力，整合时序情感演化信息。针对长文本，采用滑动窗口机制将输入分割为 $N$ 个片段，每段独立编码后通过跨段注意力融合：
$$h_{\text{text}} = \text{AttentionPool}([h_{\text{CLS}}^{(1)}, \dots, h_{\text{CLS}}^{(N)}])$$
【互动演示：缩放点积注意力 (Scaled Dot-Product Attention) 计算器】
理解 $Q$（查询）和 $K$（键）向量如何决定注意力权重是掌握 Transformer 的关键。在下方工具中，您可以手动调整二维向量 $Q$ 以及两个目标向量 $K_1$ 和 $K_2$ 的坐标，同时调节缩放因子 $d_k$。观察点积结果是如何经过缩放并由 Softmax 转化为最终分配给不同位置的注意力权重百分比的。
显示可视化演示
2.1.3 领域自适应微调策略
在下游情感识别任务中，采用动态学习率调度和层冻结策略。低层（1-6层）捕获通用语法特征，保持预训练权重不变；高层（7-12层）针对情感语义进行微调。引入对比学习损失增强类间判别性：
$$\mathcal{L}_{\text{contrast}} = -\log \frac{\sum_{k \in B} \mathbb{1}_{[k \neq i]} \exp(\text{sim}(h_i, h_k)/\tau)}{\exp(\text{sim}(h_i, h_j)/\tau)}$$
其中 $\text{sim}(\cdot, \cdot)$ 为余弦相似度，$\tau$ 为温度系数，$B$ 为当前批次样本集合。
2.2 音频模态：频谱与声学特征深度分析
音频情感特征提取经历了从手工设计特征（如MFCC、prosody）到深度表征学习的演进。自监督语音表征模型（Wav2Vec 2.0、HuBERT、WavLM）通过大规模无监督预训练，捕获了丰富的声学上下文信息。
2.2.1 自监督声学表征学习
Wav2Vec 2.0采用卷积神经网络（CNN）提取原始波形的潜在表征 $z=[z_1, z_2, \dots, z_T]$，通过Transformer网络构建上下文表征。其预训练采用对比损失（Contrastive Loss）：
$$\mathcal{L}_{\text{contrastive}} = - \sum_{t=1}^T \left[ \log \frac{\exp(\text{sim}(c_t, q_t)/\kappa)}{\sum_{\tilde{q} \in Q_t} \exp(\text{sim}(c_t, \tilde{q})/\kappa)} \right]$$
其中 $c_t$ 为上下文向量，$q_t$ 为量化后的目标表征，$Q_t$ 为负样本集合，$\kappa$ 为温度参数。
HuBERT（Hidden-Unit BERT）通过离线聚类生成伪标签，采用掩码预测损失（Masked Prediction Loss）：
$$\mathcal{L}_{\text{Hubert}} = \sum_{t \in M} \text{CrossEntropy}(p_t, \hat{y}_t)$$
其中 $M$ 为掩码位置集合，$p_t$ 为模型预测分布，$\hat{y}_t$ 为聚类标签。WavLM在HuBERT基础上引入说话人身份解耦和噪声鲁棒性训练，进一步提升了情感相关表征的纯净度。
2.2.2 时频域联合分析
梅尔频谱图（Mel-Spectrogram）将时域信号转换为时频域表示：
$$S_{\text{mel}}[t,m] = \sum_{k=0}^{K-1} |X[t,k]|^2 \cdot M[k,m]$$
其中 $X[t,k]$ 为短时傅里叶变换（STFT）系数，$M[k,m]$ 为梅尔滤波器组。采用对数压缩增强低能量成分：
$$S_{\text{log-mel}} = \log(S_{\text{mel}} + \epsilon)$$
时域特征与频域特征的融合通过双路径架构实现：一路经Wav2Vec 2.0提取高层语义表征（768维），另一路经CNN处理对数梅尔谱图（输出1920维DeepSpectrum特征）。两路特征通过加权拼接形成联合表征：
$$f_{\text{audio}} = [\lambda \cdot f_{\text{wav2vec}}; (1-\lambda) \cdot f_{\text{mel}}]$$
2.2.3 时序建模与注意力汇聚
针对变长音频序列，采用分段聚合策略。将音频划分为 $L$ 秒的非重叠窗，每窗提取特征后通过自注意力汇聚（Self-Attention Pooling）：
$$h_{\text{audio}} = \sum_{i=1}^N \beta_i f_i, \quad \beta_i = \frac{\exp(w^T f_i)}{\sum_{j=1}^N \exp(w^T f_j)}$$
其中 $w$ 为可学习的上下文向量。引入多头机制增强多尺度情感线索捕获能力。
2.3 视觉模态：CNN/Vision Transformer特征提取
面部情感识别（Facial Expression Recognition, FER）依赖于对细微面部肌肉运动的精确建模。从卷积神经网络到Vision Transformer的演进，体现了从局部特征提取到全局依赖建模的范式转变。
2.3.1 视觉Transformer架构
Vision Transformer（ViT）将图像 $I \in \mathbb{R}^{H \times W \times C}$ 划分为 $N$ 个固定大小的图像块（Patches）$x_p \in \mathbb{R}^{P^2 \cdot C}$，其中 $P$ 为块尺寸（通常为16）。通过线性投影映射到 $D$ 维嵌入空间：
$$z_0 = [x_{\text{class}}; x_p^1 E; x_p^2 E; \dots; x_p^N E] + E_{\text{pos}}$$
其中 $E \in \mathbb{R}^{(P^2 \cdot C) \times D}$ 为投影矩阵，$E_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}$ 为位置嵌入，$x_{\text{class}}$ 为分类令牌。
Transformer编码器由交替的多头注意力（MSA）和多层感知机（MLP）块构成：
$$z_l' = \text{MSA}(\text{LN}(z_{l-1})) + z_{l-1}$$
$$z_l = \text{MLP}(\text{LN}(z_l')) + z_l'$$
MSA机制计算 $h$ 个注意力头的并行表征：
$$\text{MSA}(z) = [\text{head}_1; \dots; \text{head}_h] W_O$$
$$\text{head}_i = \text{Attention}(zW_i^Q, zW_i^K, zW_i^V)$$
2.3.2 面部先验与注意力引导
针对面部情感识别，引入面部动作单元（Action Units, AUs）先验知识。通过注意力监督机制（Attention Supervision Framework, ASF），在关键面部区域（眼部、嘴部）施加高注意力权重：
$$\mathcal{L}_{\text{attention}} = - \sum_{i \in R_{\text{key}}} \log A_i$$
其中 $A_i$ 为位置 $i$ 的注意力权重，$R_{\text{key}}$ 为关键区域索引集合。混合架构（MobileNetV2-ViT）结合CNN的局部特征提取能力与Transformer的全局建模能力：MobileNetV2作为特征提取器生成局部特征图，ViT捕获长距离面部组件关系。
2.3.3 表征解耦与生成式增强
Poker Face Vision Transformer（PF-ViT）提出情感-无关特征解耦策略。假设表情面部 $x$ 由中性面部 $x_{\text{neutral}}$ 和情感组件 $x_{\text{emo}}$ 复合而成：
$$x = x_{\text{neutral}} \oplus x_{\text{emo}}$$
通过生成对抗框架，训练解码器生成对应的中性面部（Poker Face），迫使编码器捕获纯净的情感表征。解耦损失包括重构损失 $\mathcal{L}_{\text{recon}}$、对抗损失 $\mathcal{L}_{\text{adv}}$ 和身份保持损失 $\mathcal{L}_{\text{id}}$：
$$\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\text{recon}} + \lambda_2 \mathcal{L}_{\text{adv}} + \lambda_3 \mathcal{L}_{\text{id}}$$
2.4 多模态数据加载、对齐与内存优化策略
多模态情感识别的核心挑战在于处理异构数据的高维异质性、时序非对齐性以及大规模数据的高效加载。
2.4.1 分层数据加载架构
采用分层加载策略：底层为原始数据（视频帧、音频波形、文本序列），中层为预计算特征缓存（Wav2Vec特征、ViT特征、BERT特征），顶层为动态批处理采样器。针对视频数据，采用稀疏采样策略：从视频中均匀抽取 $N$ 帧而非逐帧处理，降低计算冗余：
$$\mathcal{T} = \left\{t_i \mid t_i = i \cdot \frac{T}{N}, i=0, \dots, N-1 \right\}$$
音频-文本对齐采用强制对齐（Forced Alignment）技术，基于连接时序分类（CTC）损失确定音素边界，实现词级时间戳标注。对于无时序文本的纯音频，采用滑动窗分割（Windowing）与填充（Padding）策略：固定窗长 $W$（如6秒），步长 $S$，不足部分采用前端零填充（Front Padding）：
$$x_{\text{padded}} = [\mathbf{0}_{(W-L) \times d}; x_{1:L}]$$
2.4.2 跨模态对齐机制
特征级对齐采用最优传输（Optimal Transport, OT）理论。设文本特征分布 $\mu \in \mathbb{R}^{n \times d}$，音频特征分布 $\nu \in \mathbb{R}^{m \times d}$，寻找最优耦合矩阵 $P^*$：
$$P^* = \text{argmin}_{P \in \Pi(\mu,\nu)} \langle C, P \rangle - \lambda H(P)$$
其中 $C$ 为代价矩阵（欧氏距离），$H(P)$ 为熵正则化项，$\Pi(\mu,\nu)$ 为边缘约束集合。采用Sinkhorn迭代或Greenkhorn算法求解，实现软对齐（Soft Alignment）。
注意力级对齐通过交叉注意力（Cross-Modal Attention）实现。设查询（Query）来自文本模态 $Q_t$，键值（Key-Value）来自视觉模态 $K_v, V_v$：
$$A_{t \leftarrow v} = \text{softmax}\left(\frac{Q_t K_v^T}{\sqrt{d_k}}\right)$$
$$H_{t \leftarrow v} = A_{t \leftarrow v} V_v$$
多轮循环对齐（Recurrent Alignment）通过记忆模块迭代精炼对齐质量：
$$M^{(l+1)} = \text{LayerNorm}(M^{(l)} + \text{CrossAttn}(Q^{(l)}, K^{(l)}, V^{(l)}))$$
2.4.3 内存优化与计算加速
采用梯度检查点（Gradient Checkpointing）技术，在反向传播时重计算前向激活而非存储，以时间换空间，使显存占用从 $O(L)$ 降至 $O(\sqrt{L})$（$L$ 为网络层数）。
混合精度训练（Mixed Precision Training）结合FP32主权重与FP16/FP8前向计算，通过损失缩放（Loss Scaling）防止梯度下溢：
$$\text{Loss}_{\text{scaled}} = \text{Loss} \cdot 2^{16}$$
动态批次处理（Dynamic Batching）依据序列长度分桶（Bucketing），减少填充（Padding）冗余。对于长度分布差异大的数据，采用长度感知采样器（Length-Aware Sampler）确保同批次内序列长度相近。
第二部分：结构化伪代码
Algorithm 1: 分层文本特征提取与优化
输入 (Input): 文本序列 $x=[x_1, \dots, x_T]$，预训练BERT权重 $\theta$，微调层索引集合 $F$，学习率 $\eta$
输出 (Output): 情感判别表征 $h_{\text{text}} \in \mathbb{R}^d$
function $\text{TextFeatureExtraction}(x, \theta, F)$:
-
for $t=1$ to $T$ do
-
$e_t \leftarrow \text{EmbeddingLookup}(x_t)$
-
-
$H^{(0)} \leftarrow [e_1; \dots; e_T] + E_{\text{pos}}$
-
for $l=1$ to $L$ do
-
if $l \in F$ then
-
$H^{(l)} \leftarrow \text{TransformerLayer}(H^{(l-1)}, \text{trainable=True})$
-
-
else
-
$H^{(l)} \leftarrow \text{TransformerLayer}(H^{(l-1)}, \text{trainable=False})$
-
-
-
$h_{\text{CLS}} \leftarrow H^{(L)}[0,:]$
-
$h_{\text{text}} \leftarrow \text{AttentionPool}(h_{\text{CLS}})$
-
return $h_{\text{text}}$
Algorithm 2: 自监督音频特征提取与时序汇聚
输入 (Input): 原始波形 $w \in \mathbb{R}^S$，窗长 $W$，步长 $S$，预训练Wav2Vec模型 $\phi$
输出 (Output): 固定维音频表征 $f_{\text{audio}} \in \mathbb{R}^{d_a}$
function $\text{AudioFeatureExtraction}(w, W, S, \phi)$:
-
$\mathcal{W} \leftarrow \text{Segment}(w, W, S)$
-
$\mathcal{F} \leftarrow \emptyset$
-
for $w_i \in \mathcal{W}$ do
-
$z_i \leftarrow \text{CNNFrontend}(w_i)$
-
$c_i \leftarrow \text{TransformerContext}(z_i, \phi)$
-
$f_i \leftarrow \text{MeanPool}(c_i)$
-
$\mathcal{F} \leftarrow \mathcal{F} \cup \{f_i\}$
-
-
$h_{\text{wav2vec}} \leftarrow \text{SelfAttentionPool}(\mathcal{F}, w_{\text{query}})$
-
$S_{\text{mel}} \leftarrow \text{LogMelSpectrogram}(w)$
-
$h_{\text{mel}} \leftarrow \text{CNN}(S_{\text{mel}})$
-
$f_{\text{audio}} \leftarrow \text{Concatenate}(\lambda \cdot h_{\text{wav2vec}}, (1-\lambda) \cdot h_{\text{mel}})$
-
return $f_{\text{audio}}$
Algorithm 3: 视觉Transformer特征提取与注意力引导
输入 (Input): 图像 $I \in \mathbb{R}^{H \times W \times C}$，块尺寸 $P$，ViT模型 $\psi$，关键区域掩码 $M_{\text{key}}$
输出 (Output): 面部情感表征 $f_{\text{visual}} \in \mathbb{R}^{d_v}$
function $\text{VisualFeatureExtraction}(I, P, \psi, M_{\text{key}})$:
-
$\{x_p^i\}_{i=1}^N \leftarrow \text{Patchify}(I, P)$
-
$z_0 \leftarrow [x_{\text{class}}; x_p^1 E; \dots; x_p^N E] + E_{\text{pos}}$
-
for $l=1$ to $L$ do
-
$z_l' \leftarrow \text{MSA}(\text{LayerNorm}(z_{l-1})) + z_{l-1}$
-
$z_l \leftarrow \text{MLP}(\text{LayerNorm}(z_l')) + z_l'$
-
$A_l \leftarrow \text{ExtractAttentionWeights}(\text{MSA}_l)$
-
$\mathcal{L}_{\text{attn}}^l \leftarrow -\sum_{(i,j) \in M_{\text{key}}} \log A_l[i,j]$
-
-
$f_{\text{visual}} \leftarrow \text{LayerNorm}(z_L)[0,:]$
-
return $f_{\text{visual}}, \sum_l \mathcal{L}_{\text{attn}}^l$
Algorithm 4: 多模态最优传输对齐与融合
输入 (Input): 文本特征 $H_t \in \mathbb{R}^{n \times d}$，音频特征 $H_a \in \mathbb{R}^{m \times d}$，视觉特征 $H_v \in \mathbb{R}^{k \times d}$，正则化系数 $\lambda$，最大迭代次数 $T_{\max}$
输出 (Output): 融合多模态表征 $h_{\text{fusion}} \in \mathbb{R}^{d_f}$
function $\text{MultimodalAlignmentFusion}(H_t, H_a, H_v, \lambda, T_{\max})$:
-
function $\text{SinkhornAlign}(X, Y, \lambda, T_{\max})$:
-
$C[i,j] \leftarrow \|X_i - Y_j\|_2^2$
-
$P \leftarrow \exp(-C/\lambda)$
-
$u \leftarrow \mathbf{1}_n$
-
for $\text{iter}=1$ to $T_{\max}$ do
-
$v \leftarrow \mathbf{1}_m / (P^T u + \epsilon)$
-
$u \leftarrow \mathbf{1}_n / (Pv + \epsilon)$
-
-
$P^* \leftarrow \text{diag}(u) \cdot P \cdot \text{diag}(v)$
-
$Y_{\text{aligned}} \leftarrow P^* Y$
-
return $Y_{\text{aligned}}$
-
-
$H_a' \leftarrow \text{SinkhornAlign}(H_t, H_a, \lambda, T_{\max})$
-
$H_v' \leftarrow \text{SinkhornAlign}(H_t, H_v, \lambda, T_{\max})$
-
$Q_t \leftarrow H_t W_Q$
-
$K_a \leftarrow H_a' W_K$
-
$V_a \leftarrow H_a' W_V$
-
$A_{t \leftarrow a} \leftarrow \text{softmax}\left(\frac{Q_t K_a^T}{\sqrt{d_k}}\right)$
-
$O_a \leftarrow A_{t \leftarrow a} V_a$
-
$H_{\text{fused}} \leftarrow [H_t; O_a; H_v']$
-
$h_{\text{fusion}} \leftarrow \text{TransformerEncoder}(H_{\text{fused}})[0,:]$
-
return $h_{\text{fusion}}$
第三部分：Python实现
以下四个独立脚本构成完整的多模态情感识别系统，每个脚本均可单独运行并包含可视化功能。
脚本 2.1：文本模态Transformer特征提取与优化系统
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
脚本2.1：文本模态Transformer特征提取与优化
功能说明：
1. 基于HuggingFace Transformers加载预训练BERT/RoBERTa模型
2. 实现分层冻结微调策略（底层冻结，顶层训练）
3. 集成对比学习损失函数增强类间判别性
4. 提供注意力权重可视化与特征降维展示
使用方法：
python script_2_1_text_transformer.py --text "I am very happy today!" --model bert-base-uncased
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import BertTokenizer, BertModel, RobertaTokenizer, RobertaModel
import numpy as np
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
import argparse
import warnings
warnings.filterwarnings('ignore')
# 设置中文字体支持
plt.rcParams['font.sans-serif'] = ['SimHei', 'DejaVu Sans']
plt.rcParams['axes.unicode_minus'] = False
class HierarchicalTextEncoder(nn.Module):
"""
分层文本编码器，支持选择性层冻结与对比学习
"""
def __init__(self, model_name='bert-base-uncased', freeze_layers=8, embedding_dim=768):
super(HierarchicalTextEncoder, self).__init__()
self.model_name = model_name
self.embedding_dim = embedding_dim
# 加载预训练模型
if 'roberta' in model_name.lower():
self.tokenizer = RobertaTokenizer.from_pretrained(model_name)
self.encoder = RobertaModel.from_pretrained(model_name)
else:
self.tokenizer = BertTokenizer.from_pretrained(model_name)
self.encoder = BertModel.from_pretrained(model_name)
# 分层冻结策略：低层冻结，高层可训练
self._apply_freeze_strategy(freeze_layers)
# 对比学习投影头
self.projection_head = nn.Sequential(
nn.Linear(embedding_dim, 256),
nn.ReLU(),
nn.Linear(256, 128)
)
# 注意力汇聚层
self.attention_pool = nn.Linear(embedding_dim, 1)
def _apply_freeze_strategy(self, freeze_until_layer):
"""
实施分层微调策略
参数freeze_until_layer指定冻结到哪一层（含）
"""
# 冻结词嵌入
for param in self.encoder.embeddings.parameters():
param.requires_grad = False
# 处理Transformer层（BERT-base有12层，索引0-11）
layers = self.encoder.encoder.layer
total_layers = len(layers)
for idx, layer in enumerate(layers):
if idx < freeze_until_layer:
for param in layer.parameters():
param.requires_grad = False
else:
for param in layer.parameters():
param.requires_grad = True
print(f"[INFO] 第 {idx+1}/{total_layers} 层设置为可训练")
def forward(self, texts, return_attention=False):
"""
前向传播
参数:
texts: 输入文本列表或单条文本
return_attention: 是否返回注意力权重用于可视化
返回:
features: [CLS]令牌特征
pooled: 注意力汇聚特征
projections: 对比学习投影
attentions: （可选）注意力权重
"""
if isinstance(texts, str):
texts = [texts]
# 分词处理
inputs = self.tokenizer(
texts,
return_tensors='pt',
padding=True,
truncation=True,
max_length=128,
return_dict=True
)
if torch.cuda.is_available():
inputs = {k: v.cuda() for k, v in inputs.items()}
# 获取编码器输出
outputs = self.encoder(**inputs, output_attentions=return_attention)
# 最后一层隐藏状态: [batch_size, seq_len, hidden_dim]
last_hidden = outputs.last_hidden_state
# 提取[CLS]位置特征（分类令牌位于序列首位）
cls_features = last_hidden[:, 0, :] # [batch_size, hidden_dim]
# 注意力汇聚：计算每个时间步的重要性权重
attn_weights = F.softmax(self.attention_pool(last_hidden), dim=1) # [batch_size, seq_len, 1]
pooled_features = torch.sum(attn_weights * last_hidden, dim=1) # [batch_size, hidden_dim]
# 对比学习投影
projections = F.normalize(self.projection_head(pooled_features), dim=1)
result = {
'cls_features': cls_features,
'pooled_features': pooled_features,
'projections': projections
}
if return_attention:
# 获取最后一层的注意力权重: [batch_size, num_heads, seq_len, seq_len]
result['attentions'] = outputs.attentions[-1]
return result
def contrastive_loss(self, projections, labels, temperature=0.07):
"""
监督对比学习损失函数
参数:
projections: L2归一化后的特征向量 [batch_size, proj_dim]
labels: 情感类别标签 [batch_size]
temperature: 温度系数
返回:
loss: 对比损失标量
"""
device = projections.device
batch_size = projections.size(0)
# 计算相似度矩阵
similarity_matrix = torch.matmul(projections, projections.T) / temperature
# 创建正样本掩码（同类为正样本）
labels = labels.contiguous().view(-1, 1)
mask = torch.eq(labels, labels.T).float().to(device)
# 移除自相似性
logits_mask = torch.ones_like(mask) - torch.eye(batch_size, device=device)
mask = mask * logits_mask
# 计算log_prob
exp_logits = torch.exp(similarity_matrix) * logits_mask
log_prob = similarity_matrix - torch.log(exp_logits.sum(1, keepdim=True))
# 计算每个正样本的平均log-likelihood
mask_sum = mask.sum(1)
mask_sum = torch.where(mask_sum == 0, torch.ones_like(mask_sum), mask_sum)
loss = -(mask * log_prob).sum(1) / mask_sum
return loss.mean()
def visualize_attention(text, model):
"""
可视化BERT注意力权重分布
"""
model.eval()
with torch.no_grad():
outputs = model([text], return_attention=True)
attentions = outputs['attentions'] # [1, num_heads, seq_len, seq_len]
tokens = model.tokenizer.tokenize(text)
seq_len = len(tokens) + 2 # [CLS] + tokens + [SEP]
# 取最后一层的平均注意力（所有头对[CLS]令牌的注意力）
cls_attn = attentions[0, :, 0, :].mean(0).cpu().numpy()[:seq_len]
# 创建可视化
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
# 左图：注意力热力图
im = ax1.imshow(attentions[0, :, :, :].mean(0).cpu().numpy()[:seq_len, :seq_len],
cmap='viridis', aspect='auto')
ax1.set_xticks(range(seq_len))
ax1.set_yticks(range(seq_len))
ax1.set_xticklabels(['[CLS]'] + tokens + ['[SEP]'], rotation=45, ha='right')
ax1.set_yticklabels(['[CLS]'] + tokens + ['[SEP]'])
ax1.set_title('Cross-Token Attention Heatmap (Last Layer)')
plt.colorbar(im, ax=ax1)
# 右图：[CLS]对各个token的注意力分布
x_pos = np.arange(seq_len)
ax2.bar(x_pos, cls_attn, color='steelblue', alpha=0.7)
ax2.set_xticks(x_pos)
ax2.set_xticklabels(['[CLS]'] + tokens + ['[SEP]'], rotation=45, ha='right')
ax2.set_ylabel('Attention Weight')
ax2.set_title('Attention Distribution from [CLS] Token')
ax2.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.savefig('text_attention_visualization.png', dpi=150, bbox_inches='tight')
print("[INFO] 注意力可视化已保存至 text_attention_visualization.png")
plt.show()
def visualize_tsne_embeddings(model, sample_texts, labels):
"""
使用t-SNE可视化文本特征分布
"""
model.eval()
all_features = []
with torch.no_grad():
for text in sample_texts:
outputs = model([text])
all_features.append(outputs['pooled_features'].cpu().numpy())
features = np.concatenate(all_features, axis=0)
# t-SNE降维
tsne = TSNE(n_components=2, random_state=42, perplexity=min(30, len(sample_texts)-1))
embeddings_2d = tsne.fit_transform(features)
# 可视化
plt.figure(figsize=(10, 8))
scatter = plt.scatter(
embeddings_2d[:, 0],
embeddings_2d[:, 1],
c=labels,
cmap='tab10',
s=100,
alpha=0.8,
edgecolors='white',
linewidth=2
)
plt.colorbar(scatter, label='Emotion Class')
# 添加样本标签
for i, text in enumerate(sample_texts):
plt.annotate(
text[:20] + '...',
(embeddings_2d[i, 0], embeddings_2d[i, 1]),
fontsize=8,
alpha=0.7
)
plt.title('t-SNE Visualization of Text Embeddings')
plt.xlabel('t-SNE Dimension 1')
plt.ylabel('t-SNE Dimension 2')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('text_tsne_visualization.png', dpi=150, bbox_inches='tight')
print("[INFO] t-SNE可视化已保存至 text_tsne_visualization.png")
plt.show()
def main():
parser = argparse.ArgumentParser(description='文本模态Transformer特征提取')
parser.add_argument('--text', type=str, default="I am extremely happy about this wonderful news!",
help='输入文本')
parser.add_argument('--model', type=str, default='bert-base-uncased',
help='预训练模型名称')
parser.add_argument('--freeze_layers', type=int, default=8,
help='冻结层数（BERT-base共12层）')
args = parser.parse_args()
print(f"[INIT] 加载模型: {args.model}")
print(f"[CONFIG] 冻结前 {args.freeze_layers} 层，微调后 {12-args.freeze_layers} 层")
# 初始化模型
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = HierarchicalTextEncoder(
model_name=args.model,
freeze_layers=args.freeze_layers
).to(device)
print(f"[INFO] 模型已加载至 {device}")
print(f"[INFO] 可训练参数: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}")
print(f"[INFO] 总参数量: {sum(p.numel() for p in model.parameters()):,}")
# 特征提取示例
print("\n" + "="*50)
print("特征提取演示")
print("="*50)
outputs = model([args.text])
print(f"输入文本: {args.text}")
print(f"CLS特征维度: {outputs['cls_features'].shape}")
print(f"汇聚特征维度: {outputs['pooled_features'].shape}")
print(f"投影特征维度: {outputs['projections'].shape}")
# 可视化注意力
print("\n" + "="*50)
print("生成注意力可视化")
print("="*50)
visualize_attention(args.text, model)
# t-SNE可视化（多样本）
print("\n" + "="*50)
print("生成t-SNE特征分布可视化")
print("="*50)
sample_texts = [
"I am very happy today!",
"This is so exciting and wonderful!",
"I feel sad and disappointed.",
"That is terrible news, I am angry.",
"What a surprise! I am shocked.",
"I love this amazing moment.",
"Feeling neutral about this topic.",
"This makes me furious and mad."
]
labels = [0, 0, 1, 2, 3, 0, 4, 2] # 0:喜悦, 1:悲伤, 2:愤怒, 3:惊讶, 4:中性
visualize_tsne_embeddings(model, sample_texts, labels)
print("\n[COMPLETE] 文本模态处理完成")
if __name__ == "__main__":
main()脚本 2.2：音频模态频谱与声学特征深度分析系统
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
脚本2.2：音频模态频谱与声学特征深度分析
功能说明：
1. 基于Wav2Vec 2.0/HuBERT提取自监督声学表征
2. 实现梅尔频谱图计算与CNN特征提取（DeepSpectrum）
3. 时序分段处理与注意力汇聚机制
4. 提供频谱图与特征空间可视化
使用方法：
python script_2_2_audio_spectrogram.py --audio_path sample.wav --model facebook/wav2vec2-base-960h
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import Wav2Vec2Processor, Wav2Vec2Model, HubertProcessor, HubertModel
import torchaudio
import torchaudio.transforms as T
import numpy as np
import matplotlib.pyplot as plt
import librosa
import librosa.display
from sklearn.decomposition import PCA
import argparse
import os
import warnings
warnings.filterwarnings('ignore')
class DeepAudioFeatureExtractor(nn.Module):
"""
深度音频特征提取器：结合Wav2Vec 2.0与梅尔频谱CNN
"""
def __init__(self, wav2vec_model='facebook/wav2vec2-base-960h',
n_mels=128, n_fft=400, hop_length=160, cnn_channels=[64, 128, 256]):
super(DeepAudioFeatureExtractor, self).__init__()
# 加载Wav2Vec 2.0
self.processor = Wav2Vec2Processor.from_pretrained(wav2vec_model)
self.wav2vec = Wav2Vec2Model.from_pretrained(wav2vec_model)
self.wav2vec_dim = self.wav2vec.config.hidden_size # 通常768
# 梅尔频谱参数
self.n_mels = n_mels
self.n_fft = n_fft
self.hop_length = hop_length
# 深度梅尔谱CNN (DeepSpectrum架构)
layers = []
in_channels = 1 # 单通道梅尔谱
for out_channels in cnn_channels:
layers.extend([
nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
nn.BatchNorm2d(out_channels),
nn.ReLU(),
nn.MaxPool2d(2, 2),
nn.Dropout(0.2)
])
in_channels = out_channels
self.mel_cnn = nn.Sequential(*layers)
# 计算CNN输出维度（假设输入为[batch, 1, n_mels, time_frames]）
# 经过3次池化后，空间维度减少8倍
self.cnn_output_dim = cnn_channels[-1] * (n_mels // 8) * 4 # 简化计算
# 特征融合层
self.fusion_layer = nn.Sequential(
nn.Linear(self.wav2vec_dim + cnn_channels[-1] * 4, 512),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(512, 256)
)
# 自注意力汇聚
self.attention_query = nn.Parameter(torch.randn(1, 256))
self.attention_layer = nn.Linear(256, 1)
# 融合权重（可学习）
self.fusion_weight = nn.Parameter(torch.tensor([0.6])) # 默认Wav2Vec权重较高
def extract_wav2vec_features(self, waveform, sample_rate=16000):
"""
提取Wav2Vec 2.0特征
"""
# 重采样到16kHz（Wav2Vec要求）
if sample_rate != 16000:
resampler = T.Resample(orig_freq=sample_rate, new_freq=16000)
waveform = resampler(waveform)
# 确保输入维度正确 [batch, samples]
if waveform.dim() == 1:
waveform = waveform.unsqueeze(0)
# 使用processor处理
inputs = self.processor(
waveform.squeeze().numpy(),
sampling_rate=16000,
return_tensors="pt",
padding=True
)
if torch.cuda.is_available():
inputs = {k: v.cuda() for k, v in inputs.items()}
with torch.no_grad():
outputs = self.wav2vec(**inputs)
# 最后一层隐藏状态: [batch, time, hidden]
features = outputs.last_hidden_state
# 均值汇聚（可替换为注意力汇聚）
wav2vec_feat = features.mean(dim=1) # [batch, hidden]
return wav2vec_feat, features # 返回汇聚特征和时序特征
def extract_mel_features(self, waveform, sample_rate=16000):
"""
提取梅尔频谱CNN特征
"""
if sample_rate != 16000:
resampler = T.Resample(orig_freq=sample_rate, new_freq=16000)
waveform = resampler(waveform)
# 计算梅尔频谱
mel_spectrogram = T.MelSpectrogram(
sample_rate=16000,
n_fft=self.n_fft,
hop_length=self.hop_length,
n_mels=self.n_mels,
power=2.0
)
if torch.cuda.is_available():
mel_spectrogram = mel_spectrogram.cuda()
mel_spec = mel_spectrogram(waveform) # [batch, n_mels, time]
# 对数压缩
mel_spec_db = T.AmplitudeToDB(st_ref=1.0, top_db=80)(mel_spec)
# 归一化
mel_spec_db = (mel_spec_db - mel_spec_db.mean()) / (mel_spec_db.std() + 1e-8)
# 添加通道维度 [batch, 1, n_mels, time]
mel_spec_db = mel_spec_db.unsqueeze(1)
# CNN特征提取
cnn_features = self.mel_cnn(mel_spec_db) # [batch, channels, height, width]
# 全局平均池化
cnn_feat = F.adaptive_avg_pool2d(cnn_features, (1, 1))
cnn_feat = cnn_feat.view(cnn_feat.size(0), -1) # [batch, features]
return cnn_feat, mel_spec_db.squeeze(1)
def attention_pooling(self, features):
"""
自注意力汇聚
参数: features [batch, seq_len, dim]
"""
# 计算注意力权重
attn_scores = self.attention_layer(features).squeeze(-1) # [batch, seq_len]
attn_weights = F.softmax(attn_scores, dim=1)
# 加权求和
pooled = torch.sum(attn_weights.unsqueeze(-1) * features, dim=1)
return pooled, attn_weights
def forward(self, waveform, sample_rate=16000):
"""
完整前向传播，融合Wav2Vec与梅尔特征
"""
# 提取双路特征
wav2vec_feat, wav2vec_seq = self.extract_wav2vec_features(waveform, sample_rate)
cnn_feat, mel_spec = self.extract_mel_features(waveform, sample_rate)
# 特征投影到相同维度
batch_size = waveform.size(0)
# 确保维度匹配
if cnn_feat.size(1) < wav2vec_feat.size(1):
cnn_feat = F.pad(cnn_feat, (0, wav2vec_feat.size(1) - cnn_feat.size(1)))
# 加权融合
w = torch.sigmoid(self.fusion_weight)
fused = torch.cat([
w * wav2vec_feat,
(1 - w) * cnn_feat[:, :wav2vec_feat.size(1)]
], dim=1)
# 通过融合层
output = self.fusion_layer(fused)
return {
'fused_features': output,
'wav2vec_features': wav2vec_feat,
'mel_cnn_features': cnn_feat[:, :256],
'mel_spectrogram': mel_spec,
'fusion_weight': w.item()
}
def visualize_mel_spectrogram(mel_spec, title="Mel Spectrogram"):
"""
可视化梅尔频谱图
"""
plt.figure(figsize=(12, 4))
if torch.is_tensor(mel_spec):
mel_spec = mel_spec.cpu().numpy()
if mel_spec.ndim == 3:
mel_spec = mel_spec[0] # 取batch第一个
librosa.display.specshow(
mel_spec,
y_axis='mel',
x_axis='time',
sr=16000,
hop_length=160,
cmap='viridis'
)
plt.colorbar(format='%+2.0f dB')
plt.title(title)
plt.tight_layout()
plt.savefig('mel_spectrogram.png', dpi=150, bbox_inches='tight')
print("[INFO] 梅尔频谱图已保存至 mel_spectrogram.png")
plt.show()
def visualize_feature_space(features_list, labels, method='pca'):
"""
可视化特征空间分布
"""
if torch.is_tensor(features_list[0]):
features = torch.stack(features_list).cpu().numpy()
else:
features = np.array(features_list)
# 降维
if method == 'pca':
reducer = PCA(n_components=2)
embeddings = reducer.fit_transform(features)
title = 'PCA Projection of Audio Features'
else:
from sklearn.manifold import TSNE
reducer = TSNE(n_components=2, random_state=42)
embeddings = reducer.fit_transform(features)
title = 't-SNE Projection of Audio Features'
# 可视化
plt.figure(figsize=(10, 8))
scatter = plt.scatter(
embeddings[:, 0],
embeddings[:, 1],
c=labels,
cmap='tab10',
s=150,
alpha=0.7,
edgecolors='white',
linewidth=2
)
plt.colorbar(scatter, label='Emotion Class')
plt.title(title)
plt.xlabel('Component 1')
plt.ylabel('Component 2')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig(f'audio_{method}_visualization.png', dpi=150, bbox_inches='tight')
print(f"[INFO] 特征空间可视化已保存至 audio_{method}_visualization.png")
plt.show()
def generate_synthetic_audio(duration=3.0, sample_rate=16000, emotion='happy'):
"""
生成合成音频用于演示（实际使用时应加载真实音频文件）
"""
t = np.linspace(0, duration, int(sample_rate * duration))
if emotion == 'happy':
# 高频、快速调制
freq = 300 + 100 * np.sin(2 * np.pi * 5 * t)
audio = np.sin(2 * np.pi * freq * t) * (0.5 + 0.5 * np.sin(2 * np.pi * 3 * t))
elif emotion == 'sad':
# 低频、缓慢
freq = 150
audio = np.sin(2 * np.pi * freq * t) * np.exp(-t * 0.5)
elif emotion == 'angry':
# 噪声、高频
audio = np.random.randn(len(t)) * 0.3 + np.sin(2 * np.pi * 400 * t)
else: # neutral
freq = 200
audio = np.sin(2 * np.pi * freq * t)
# 添加轻微噪声模拟真实场景
audio += np.random.randn(len(audio)) * 0.01
audio = audio / np.max(np.abs(audio))
return torch.tensor(audio, dtype=torch.float32).unsqueeze(0)
def main():
parser = argparse.ArgumentParser(description='音频模态深度特征提取')
parser.add_argument('--audio_path', type=str, default=None,
help='音频文件路径（若不提供则生成合成音频）')
parser.add_argument('--model', type=str, default='facebook/wav2vec2-base-960h',
help='Wav2Vec 2.0模型名称')
parser.add_argument('--duration', type=float, default=3.0,
help='合成音频时长（秒）')
args = parser.parse_args()
print(f"[INIT] 初始化音频特征提取器")
print(f"[CONFIG] 使用模型: {args.model}")
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = DeepAudioFeatureExtractor(wav2vec_model=args.model).to(device)
model.eval()
print(f"[INFO] 模型已加载至 {device}")
print(f"[INFO] 可训练参数: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}")
# 加载或生成音频
if args.audio_path and os.path.exists(args.audio_path):
print(f"[LOAD] 加载音频文件: {args.audio_path}")
waveform, sample_rate = torchaudio.load(args.audio_path)
waveform = waveform.mean(dim=0, keepdim=True) # 转单声道
else:
print(f"[DEMO] 生成合成音频用于演示（时长: {args.duration}s）")
waveform = generate_synthetic_audio(duration=args.duration)
sample_rate = 16000
if torch.cuda.is_available():
waveform = waveform.cuda()
print(f"[INFO] 音频形状: {waveform.shape}, 采样率: {sample_rate}")
# 特征提取
print("\n" + "="*50)
print("执行深度特征提取")
print("="*50)
with torch.no_grad():
outputs = model(waveform, sample_rate)
print(f"融合特征维度: {outputs['fused_features'].shape}")
print(f"Wav2Vec特征维度: {outputs['wav2vec_features'].shape}")
print(f"Mel-CNN特征维度: {outputs['mel_cnn_features'].shape}")
print(f"融合权重 (Wav2Vec占比): {outputs['fusion_weight']:.3f}")
# 可视化梅尔频谱
print("\n" + "="*50)
print("生成梅尔频谱可视化")
print("="*50)
visualize_mel_spectrogram(outputs['mel_spectrogram'])
# 多类特征空间可视化
print("\n" + "="*50)
print("生成多类别特征空间可视化")
print("="*50)
# 生成不同情感的合成音频进行可视化
emotions = ['happy', 'sad', 'angry', 'neutral', 'happy', 'sad']
labels = [0, 1, 2, 3, 0, 1]
all_features = []
for emotion in emotions:
wave = generate_synthetic_audio(duration=2.0, emotion=emotion)
if torch.cuda.is_available():
wave = wave.cuda()
with torch.no_grad():
feat = model(wave)['fused_features']
all_features.append(feat.cpu())
visualize_feature_space(all_features, labels, method='pca')
print("\n[COMPLETE] 音频模态处理完成")
if __name__ == "__main__":
main()脚本 2.3：视觉模态CNN/Vision Transformer特征提取系统
Python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
脚本2.3：视觉模态CNN/Vision Transformer特征提取
功能说明：
1. 基于ViT（Vision Transformer）的面部情感特征提取
2. 实现注意力监督机制（ASF）引导关键面部区域关注
3. 支持Poker Face生成与情感-中性表征解耦
4. 提供注意力热力图与面部关键点可视化
使用方法：
python script_2_3_visual_vit.py --image_path face.jpg --model google/vit-base-patch16-224
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import ViTImageProcessor, ViTModel, ViTConfig
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
import cv2
from sklearn.manifold import TSNE
import argparse
import os
import warnings
warnings.filterwarnings('ignore')
class AttentionGuidedViT(nn.Module):
"""
注意力引导的Vision Transformer，支持面部情感识别
集成注意力监督机制（ASF）与表征解耦
"""
def __init__(self, model_name='google/vit-base-patch16-224', num_classes=7,
patch_size=16, img_size=224):
super(AttentionGuidedViT, self).__init__()
# 加载预训练ViT
self.processor = ViTImageProcessor.from_pretrained(model_name)
self.vit = ViTModel.from_pretrained(model_name)
self.config = self.vit.config
self.hidden_size = self.config.hidden_size # 768
self.patch_size = patch_size
self.num_patches = (img_size // patch_size) ** 2 # 196 for 224/16
self.img_size = img_size
# 分类头
self.classifier = nn.Sequential(
nn.LayerNorm(self.hidden_size),
nn.Dropout(0.1),
nn.Linear(self.hidden_size, 256),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(256, num_classes)
)
# 注意力监督权重（关键面部区域：眼部、嘴部）
self.register_buffer('attention_mask', torch.zeros(self.num_patches))
# 情感解耦模块（PF-ViT风格）
self.emotion_decoder = nn.Sequential(
nn.Linear(self.hidden_size, 512),
nn.ReLU(),
nn.Linear(512, self.num_patches * 3) # 生成RGB扑克脸
)
# 身份保持判别器（简化版）
self.identity_projector = nn.Linear(self.hidden_size, 128)
def set_attention_mask(self, key_regions):
"""
设置关键区域掩码用于注意力监督
key_regions: 关键区域patch索引列表
"""
mask = torch.zeros(self.num_patches)
for idx in key_regions:
if 0 <= idx < self.num_patches:
mask[idx] = 1.0
self.attention_mask = mask
def extract_attention_weights(self, outputs):
"""
从ViT输出中提取注意力权重
"""
# attentions: tuple of [batch, num_heads, seq_len, seq_len]
# 共12层，取最后一层
last_attentions = outputs.attentions[-1] if outputs.attentions else None
if last_attentions is None:
return None
# 对所有头取平均 [batch, seq_len, seq_len]
avg_attention = last_attentions.mean(dim=1)
# 提取[CLS]令牌对所有patch的注意力 [batch, num_patches]
cls_attention = avg_attention[:, 0, 1:] # 忽略[CLS]自身
return cls_attention
def compute_attention_supervision_loss(self, cls_attention):
"""
计算注意力监督损失（ASF）
鼓励模型关注关键面部区域
"""
if self.attention_mask.sum() == 0:
return torch.tensor(0.0).to(cls_attention.device)
# 归一化注意力权重
cls_attention = cls_attention / (cls_attention.sum(dim=1, keepdim=True) + 1e-8)
# 关键区域掩码扩展到batch
mask = self.attention_mask.unsqueeze(0).expand_as(cls_attention)
# 交叉熵风格的损失：鼓励关键区域获得高注意力
# 使用负对数似然
loss = -torch.sum(mask * torch.log(cls_attention + 1e-8)) / (mask.sum() + 1e-8)
return loss
def disentangle_representation(self, cls_feature):
"""
情感-中性表征解耦
返回情感组件和中性重构
"""
# 生成伪中性面部（Poker Face）
reconstructed_neutral = self.emotion_decoder(cls_feature)
reconstructed_neutral = reconstructed_neutral.view(-1, self.num_patches, 3)
# 情感残差（原始特征与中性特征的差异）
emotion_residual = cls_feature # 简化为直接使用CLS特征
return emotion_residual, reconstructed_neutral
def forward(self, images, return_attention=False, apply_supervision=False):
"""
前向传播
参数:
images: PIL Image列表或张量
return_attention: 是否返回注意力图
apply_supervision: 是否应用注意力监督损失
返回:
logits: 分类 logits
features: 特征向量
attention_loss: （可选）监督损失
attention_maps: （可选）注意力图
"""
# 预处理
if isinstance(images, list) and isinstance(images[0], Image.Image):
inputs = self.processor(images, return_tensors="pt")
elif isinstance(images, Image.Image):
inputs = self.processor([images], return_tensors="pt")
else:
inputs = {'pixel_values': images}
if torch.cuda.is_available():
inputs = {k: v.cuda() for k, v in inputs.items()}
# ViT前向
outputs = self.vit(**inputs, output_attentions=True)
# [CLS]特征
cls_feature = outputs.last_hidden_state[:, 0, :] # [batch, hidden]
# 分类
logits = self.classifier(cls_feature)
result = {
'logits': logits,
'features': cls_feature,
'all_hidden_states': outputs.last_hidden_state
}
# 提取注意力
if return_attention or apply_supervision:
cls_attention = self.extract_attention_weights(outputs)
result['attention_maps'] = cls_attention
# 计算监督损失
if apply_supervision and cls_attention is not None:
attn_loss = self.compute_attention_supervision_loss(cls_attention)
result['attention_loss'] = attn_loss
# 表征解耦
emotion_feat, neutral_recon = self.disentangle_representation(cls_feature)
result['emotion_features'] = emotion_feat
result['neutral_reconstruction'] = neutral_recon
return result
def get_attention_heatmap(self, image, attention_weights):
"""
生成注意力热力图叠加图像
"""
if torch.is_tensor(attention_weights):
attention_weights = attention_weights.cpu().numpy()
if attention_weights.ndim == 2:
attention_weights = attention_weights[0] # 取batch第一个
# 将attention reshape为2D网格 (14x14 for 224/16)
grid_size = int(np.sqrt(len(attention_weights)))
attention_map = attention_weights.reshape(grid_size, grid_size)
# 上采样到原图尺寸
attention_map = cv2.resize(
attention_map,
(self.img_size, self.img_size),
interpolation=cv2.INTER_CUBIC
)
# 归一化到0-1
attention_map = (attention_map - attention_map.min()) / (attention_map.max() - attention_map.min() + 1e-8)
return attention_map
def visualize_attention_overlay(image, model, save_path='attention_overlay.png'):
"""
在原图上叠加注意力热力图
"""
model.eval()
with torch.no_grad():
outputs = model(image, return_attention=True)
attention = outputs['attention_maps'] # [1, num_patches]
# 生成热力图
heatmap = model.get_attention_heatmap(image, attention)
# 转换PIL图像为numpy数组
if isinstance(image, Image.Image):
img_array = np.array(image.convert('RGB'))
img_array = cv2.resize(img_array, (model.img_size, model.img_size))
else:
img_array = image
# 叠加热力图
heatmap_color = cv2.applyColorMap(np.uint8(255 * heatmap), cv2.COLORMAP_JET)
heatmap_color = cv2.cvtColor(heatmap_color, cv2.COLOR_BGR2RGB)
overlay = cv2.addWeighted(img_array, 0.6, heatmap_color, 0.4, 0)
# 可视化
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
axes[0].imshow(img_array)
axes[0].set_title('Original Image')
axes[0].axis('off')
axes[1].imshow(heatmap, cmap='jet')
axes[1].set_title('Attention Heatmap')
axes[1].axis('off')
axes[2].imshow(overlay)
axes[2].set_title('Overlay')
axes[2].axis('off')
plt.tight_layout()
plt.savefig(save_path, dpi=150, bbox_inches='tight')
print(f"[INFO] 注意力可视化已保存至 {save_path}")
plt.show()
return overlay
def visualize_patch_embeddings(model, images_list, labels):
"""
使用t-SNE可视化ViT的patch嵌入分布
"""
model.eval()
all_cls_features = []
with torch.no_grad():
for img in images_list:
outputs = model(img)
all_cls_features.append(outputs['features'].cpu())
features = torch.cat(all_cls_features, dim=0).numpy()
# t-SNE降维
tsne = TSNE(n_components=2, random_state=42, perplexity=min(30, len(features)-1))
embeddings = tsne.fit_transform(features)
# 可视化
plt.figure(figsize=(10, 8))
scatter = plt.scatter(
embeddings[:, 0],
embeddings[:, 1],
c=labels,
cmap='tab10',
s=200,
alpha=0.7,
edgecolors='white',
linewidth=2
)
plt.colorbar(scatter, label='Emotion Class')
plt.title('t-SNE Visualization of ViT [CLS] Embeddings')
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('vit_embeddings_tsne.png', dpi=150, bbox_inches='tight')
print("[INFO] 特征嵌入可视化已保存至 vit_embeddings_tsne.png")
plt.show()
def generate_synthetic_face(size=(224, 224), emotion='happy'):
"""
生成合成面部图像用于演示（使用简单几何形状模拟）
"""
img = np.ones((*size, 3), dtype=np.uint8) * 240 # 浅色背景
center = (size[0] // 2, size[1] // 2)
# 脸轮廓
cv2.circle(img, center, 80, (200, 200, 200), -1)
# 眼睛
eye_left = (center[0] - 30, center[1] - 20)
eye_right = (center[0] + 30, center[1] - 20)
cv2.circle(img, eye_left, 8, (50, 50, 50), -1)
cv2.circle(img, eye_right, 8, (50, 50, 50), -1)
# 根据情感绘制嘴部
if emotion == 'happy':
# 微笑
cv2.ellipse(img, (center[0], center[1] + 30), (40, 20), 0, 0, 180, (100, 100, 100), 3)
elif emotion == 'sad':
# 悲伤
cv2.ellipse(img, (center[0], center[1] + 40), (30, 15), 0, 180, 360, (100, 100, 100), 3)
elif emotion == 'angry':
# 愤怒（皱眉）
cv2.line(img, (center[0] - 40, center[1] + 30), (center[0] + 40, center[1] + 30), (100, 100, 100), 3)
# 眉毛下斜
cv2.line(img, (center[0] - 40, center[1] - 40), (center[0] - 20, center[1] - 25), (80, 80, 80), 4)
cv2.line(img, (center[0] + 40, center[1] - 40), (center[0] + 20, center[1] - 25), (80, 80, 80), 4)
else: # neutral
cv2.line(img, (center[0] - 30, center[1] + 35), (center[0] + 30, center[1] + 35), (100, 100, 100), 3)
return Image.fromarray(img)
def main():
parser = argparse.ArgumentParser(description='视觉模态ViT特征提取')
parser.add_argument('--image_path', type=str, default=None,
help='面部图像路径（若不提供则生成合成图像）')
parser.add_argument('--model', type=str, default='google/vit-base-patch16-224',
help='ViT预训练模型')
parser.add_argument('--emotion', type=str, default='happy',
choices=['happy', 'sad', 'angry', 'neutral', 'fear', 'disgust', 'surprise'],
help='生成图像的情感类型')
args = parser.parse_args()
print(f"[INIT] 初始化ViT视觉特征提取器")
print(f"[CONFIG] 模型: {args.model}")
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = AttentionGuidedViT(model_name=args.model, num_classes=7).to(device)
model.eval()
# 设置关键区域掩码（眼部和嘴部对应的patch索引）
# 对于14x14的网格，眼睛大约在[74,75,76,88,89,90]，嘴部在[160-177]
key_regions = [74, 75, 76, 88, 89, 90, 162, 163, 164, 165, 176, 177, 178, 179]
model.set_attention_mask(key_regions)
print(f"[INFO] 模型已加载至 {device}")
print(f"[INFO] 关键区域掩码已设置（{len(key_regions)}个patch）")
# 加载或生成图像
if args.image_path and os.path.exists(args.image_path):
print(f"[LOAD] 加载图像: {args.image_path}")
image = Image.open(args.image_path).convert('RGB')
# 调整尺寸
image = image.resize((224, 224))
else:
print(f"[DEMO] 生成合成{args.emotion}面部图像")
image = generate_synthetic_face(emotion=args.emotion)
# 特征提取
print("\n" + "="*50)
print("执行ViT特征提取")
print("="*50)
with torch.no_grad():
outputs = model(image, return_attention=True, apply_supervision=True)
print(f"分类Logits: {outputs['logits'].shape}")
print(f"[CLS]特征维度: {outputs['features'].shape}")
print(f"注意力损失: {outputs.get('attention_loss', 0):.4f}")
# 注意力可视化
print("\n" + "="*50)
print("生成注意力可视化")
print("="*50)
visualize_attention_overlay(image, model)
# 多类别特征空间可视化
print("\n" + "="*50)
print("生成特征空间分布可视化")
print("="*50)
emotions = ['happy', 'sad', 'angry', 'neutral', 'happy', 'sad']
labels = [0, 1, 2, 3, 0, 1]
images = [generate_synthetic_face(emotion=e) for e in emotions]
visualize_patch_embeddings(model, images, labels)
print("\n[COMPLETE] 视觉模态处理完成")
if __name__ == "__main__":
main()脚本 2.4：多模态数据加载、对齐与内存优化系统
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
脚本2.4：多模态数据加载、对齐与内存优化
功能说明：
1. 实现分层数据加载与预计算特征缓存机制
2. 基于最优传输（Sinkhorn）的跨模态特征对齐
3. 动态批次处理（Dynamic Bucketing）与内存优化
4. 多模态融合与可视化分析
使用方法：
python script_2_4_multimodal_alignment.py --mode alignment --batch_size 4
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader, Sampler
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import cdist
import argparse
import warnings
warnings.filterwarnings('ignore')
# 设置随机种子确保可复现性
torch.manual_seed(42)
np.random.seed(42)
class OptimalTransportAlignment(nn.Module):
"""
基于最优传输理论的特征对齐模块
实现Sinkhorn算法进行软对齐
"""
def __init__(self, d_in=256, d_out=256, reg=0.1, max_iter=100):
super(OptimalTransportAlignment, self).__init__()
self.d_in = d_in
self.d_out = d_out
self.reg = reg # 熵正则化系数
self.max_iter = max_iter
# 特征投影层（统一维度）
self.text_proj = nn.Linear(768, d_out) # BERT 768维
self.audio_proj = nn.Linear(256, d_out) # Wav2Vec融合后256维
self.visual_proj = nn.Linear(768, d_out) # ViT 768维
# 交叉注意力参数
self.query_proj = nn.Linear(d_out, d_out)
self.key_proj = nn.Linear(d_out, d_out)
self.value_proj = nn.Linear(d_out, d_out)
self.scale = d_out ** -0.5
def sinkhorn(self, cost_matrix):
"""
Sinkhorn-Knopp算法求解最优传输
参数:
cost_matrix: [batch, n, m] 代价矩阵
返回:
P: [batch, n, m] 最优传输计划
"""
batch_size, n, m = cost_matrix.shape
device = cost_matrix.device
# 初始化
K = torch.exp(-cost_matrix / self.reg)
u = torch.ones(batch_size, n, device=device) / n
v = torch.ones(batch_size, m, device=device) / m
# 迭代求解
for _ in range(self.max_iter):
u = 1.0 / (torch.matmul(K, v.unsqueeze(-1)).squeeze(-1) + 1e-8)
v = 1.0 / (torch.matmul(K.transpose(1, 2), u.unsqueeze(-1)).squeeze(-1) + 1e-8)
# 计算传输计划
P = torch.matmul(torch.diag_embed(u), torch.matmul(K, torch.diag_embed(v)))
return P
def compute_cost_matrix(self, source, target):
"""
计算批量代价矩阵（欧氏距离）
"""
# source: [batch, n, d], target: [batch, m, d]
# 输出: [batch, n, m]
# 使用广播计算 pairwise distance
# source[:, :, None, :] - target[:, None, :, :]
diff = source.unsqueeze(2) - target.unsqueeze(1)
cost = torch.sum(diff ** 2, dim=-1)
return cost
def cross_modal_attention(self, query, key, value):
"""
多头交叉注意力
"""
Q = self.query_proj(query)
K = self.key_proj(key)
V = self.value_proj(value)
# 计算注意力分数
attn_scores = torch.matmul(Q, K.transpose(-2, -1)) * self.scale
attn_weights = F.softmax(attn_scores, dim=-1)
output = torch.matmul(attn_weights, V)
return output, attn_weights
def forward(self, text_feat, audio_feat, visual_feat, return_coupling=False):
"""
多模态对齐与融合
参数:
text_feat: [batch, seq_len_t, 768] 或 [batch, 768]
audio_feat: [batch, seq_len_a, 256] 或 [batch, 256]
visual_feat: [batch, seq_len_v, 768] 或 [batch, 768]
返回:
fused: [batch, d_out] 融合特征
alignments: 对齐信息字典
"""
batch_size = text_feat.size(0)
# 统一维度：如果是2D输入（已汇聚），添加序列维度
if text_feat.dim() == 2:
text_feat = text_feat.unsqueeze(1)
if audio_feat.dim() == 2:
audio_feat = audio_feat.unsqueeze(1)
if visual_feat.dim() == 2:
visual_feat = visual_feat.unsqueeze(1)
# 投影到统一空间
text_proj = self.text_proj(text_feat) # [batch, n_t, d_out]
audio_proj = self.audio_proj(audio_feat) # [batch, n_a, d_out]
visual_proj = self.visual_proj(visual_feat) # [batch, n_v, d_out]
# 基于文本对齐音频（以文本为锚点）
cost_ta = self.compute_cost_matrix(text_proj, audio_proj)
P_ta = self.sinkhorn(cost_ta) # [batch, n_t, n_a]
audio_aligned = torch.matmul(P_ta, audio_proj) # [batch, n_t, d_out]
# 基于文本对齐视觉
cost_tv = self.compute_cost_matrix(text_proj, visual_proj)
P_tv = self.sinkhorn(cost_tv)
visual_aligned = torch.matmul(P_tv, visual_proj)
# 交叉注意力融合
# 文本作为Query，对齐后的音频视觉作为KV
kv = torch.cat([audio_aligned, visual_aligned], dim=1) # [batch, n_t*2, d_out]
fused_seq, attn_weights = self.cross_modal_attention(text_proj, kv, kv)
# 全局平均汇聚
fused = fused_seq.mean(dim=1) # [batch, d_out]
result = {
'fused_features': fused,
'text_features': text_proj.mean(dim=1),
'audio_aligned': audio_aligned.mean(dim=1),
'visual_aligned': visual_aligned.mean(dim=1)
}
if return_coupling:
result['coupling_ta'] = P_ta
result['coupling_tv'] = P_tv
result['attention_weights'] = attn_weights
return result
class LengthAwareSampler(Sampler):
"""
长度感知采样器，减少填充padding
将相似长度的样本分到同一批次
"""
def __init__(self, lengths, batch_size, shuffle=True):
self.lengths = lengths
self.batch_size = batch_size
self.shuffle = shuffle
# 按长度分桶
self.buckets = {}
for idx, length in enumerate(lengths):
bucket_id = length // 50 # 每50个单位一个桶
if bucket_id not in self.buckets:
self.buckets[bucket_id] = []
self.buckets[bucket_id].append(idx)
def __iter__(self):
# 打乱每个桶
if self.shuffle:
for bucket in self.buckets.values():
np.random.shuffle(bucket)
# 合并所有桶
all_indices = []
for bucket_id in sorted(self.buckets.keys()):
all_indices.extend(self.buckets[bucket_id])
# 分批
batches = []
current_batch = []
for idx in all_indices:
current_batch.append(idx)
if len(current_batch) >= self.batch_size:
batches.append(current_batch)
current_batch = []
if current_batch:
batches.append(current_batch)
if self.shuffle:
np.random.shuffle(batches)
for batch in batches:
yield batch
def __len__(self):
return (len(self.lengths) + self.batch_size - 1) // self.batch_size
class MultimodalDataset(Dataset):
"""
多模态数据集模拟，支持分层特征缓存
"""
def __init__(self, num_samples=100, cache_features=True):
self.num_samples = num_samples
self.cache_features = cache_features
# 模拟不同长度的序列（音频长度变化大）
self.text_lengths = np.random.randint(10, 50, num_samples)
self.audio_lengths = np.random.randint(100, 500, num_samples) # 帧数
self.visual_lengths = np.full(num_samples, 49) # 7x7=49个patch
# 模拟预计算特征缓存
if cache_features:
self.cached_text = [torch.randn(768) for _ in range(num_samples)]
self.cached_audio = [torch.randn(256) for _ in range(num_samples)]
self.cached_visual = [torch.randn(768) for _ in range(num_samples)]
# 标签
self.labels = np.random.randint(0, 7, num_samples)
def __len__(self):
return self.num_samples
def __getitem__(self, idx):
item = {
'text': self.cached_text[idx] if self.cache_features else torch.randn(768),
'audio': self.cached_audio[idx] if self.cache_features else torch.randn(256),
'visual': self.cached_visual[idx] if self.cache_features else torch.randn(768),
'label': self.labels[idx],
'text_len': self.text_lengths[idx],
'audio_len': self.audio_lengths[idx],
'visual_len': self.visual_lengths[idx]
}
return item
class GradientCheckpointedFusion(nn.Module):
"""
带梯度检查点的融合层，节省显存
"""
def __init__(self, d_model=256, num_layers=4):
super(GradientCheckpointedFusion, self).__init__()
self.layers = nn.ModuleList([
nn.TransformerEncoderLayer(d_model=d_model, nhead=8, dim_feedforward=1024)
for _ in range(num_layers)
])
self.checkpoint_enabled = True
def forward(self, x):
for layer in self.layers:
if self.training and self.checkpoint_enabled:
# 使用梯度检查点
x = torch.utils.checkpoint.checkpoint(layer, x)
else:
x = layer(x)
return x
def visualize_optimal_transport(coupling_matrix, source_names=None, target_names=None, title="Optimal Transport Plan"):
"""
可视化最优传输计划矩阵
"""
if torch.is_tensor(coupling_matrix):
coupling_matrix = coupling_matrix.cpu().numpy()
if coupling_matrix.ndim == 3:
coupling_matrix = coupling_matrix[0] # 取batch第一个
plt.figure(figsize=(10, 8))
im = plt.imshow(coupling_matrix, cmap='YlOrRd', aspect='auto')
plt.colorbar(im, label='Transport Mass')
if source_names:
plt.yticks(range(len(source_names)), source_names)
if target_names:
plt.xticks(range(len(target_names)), target_names, rotation=45, ha='right')
plt.title(title)
plt.ylabel('Source Distribution (Text)')
plt.xlabel('Target Distribution (Audio/Visual)')
plt.tight_layout()
plt.savefig('optimal_transport_plan.png', dpi=150, bbox_inches='tight')
print(f"[INFO] 传输计划可视化已保存至 optimal_transport_plan.png")
plt.show()
def visualize_modality_alignment(features_dict, labels, method='tsne'):
"""
可视化对齐后的多模态特征分布
"""
# 解包特征
text_feats = features_dict['text_features'].cpu().numpy()
audio_feats = features_dict['audio_aligned'].cpu().numpy()
visual_feats = features_dict['visual_aligned'].cpu().numpy()
fused_feats = features_dict['fused_features'].cpu().numpy()
# 合并用于降维
all_feats = np.concatenate([text_feats, audio_feats, visual_feats, fused_feats], axis=0)
# 降维
if method == 'tsne':
from sklearn.manifold import TSNE
reducer = TSNE(n_components=2, random_state=42, perplexity=30)
else:
from sklearn.decomposition import PCA
reducer = PCA(n_components=2)
embeddings = reducer.fit_transform(all_feats)
# 分离各模态
n = len(text_feats)
text_emb = embeddings[:n]
audio_emb = embeddings[n:2*n]
visual_emb = embeddings[2*n:3*n]
fused_emb = embeddings[3*n:]
# 可视化
plt.figure(figsize=(12, 10))
colors = plt.cm.tab10(labels)
# 绘制各模态
plt.scatter(text_emb[:, 0], text_emb[:, 1], c=colors, marker='o', s=150,
alpha=0.6, label='Text', edgecolors='white', linewidth=1.5)
plt.scatter(audio_emb[:, 0], audio_emb[:, 1], c=colors, marker='s', s=150,
alpha=0.6, label='Audio (Aligned)', edgecolors='white', linewidth=1.5)
plt.scatter(visual_emb[:, 0], visual_emb[:, 1], c=colors, marker='^', s=150,
alpha=0.6, label='Visual (Aligned)', edgecolors='white', linewidth=1.5)
plt.scatter(fused_emb[:, 0], fused_emb[:, 1], c=colors, marker='*', s=300,
alpha=0.8, label='Fused', edgecolors='black', linewidth=2)
plt.legend(loc='upper right', fontsize=11)
plt.title(f'Multimodal Feature Alignment Visualization ({method.upper()})')
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig(f'multimodal_alignment_{method}.png', dpi=150, bbox_inches='tight')
print(f"[INFO] 多模态对齐可视化已保存至 multimodal_alignment_{method}.png")
plt.show()
def demonstrate_memory_optimization():
"""
演示内存优化技术效果
"""
print("\n" + "="*60)
print("内存优化技术演示")
print("="*60)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
# 模拟大模型
model_no_checkpoint = nn.TransformerEncoder(
nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048),
num_layers=12
).to(device)
model_with_checkpoint = GradientCheckpointedFusion(d_model=512, num_layers=12).to(device)
model_with_checkpoint.checkpoint_enabled = True
# 模拟输入
x = torch.randn(4, 100, 512, requires_grad=True).to(device)
# 标准模式
print("[TEST] 标准前向传播（无梯度检查点）...")
try:
torch.cuda.empty_cache()
start_mem = torch.cuda.memory_allocated(device) if torch.cuda.is_available() else 0
output1 = model_no_checkpoint(x)
loss1 = output1.sum()
loss1.backward()
end_mem = torch.cuda.memory_allocated(device) if torch.cuda.is_available() else 0
print(f" 显存占用: {(end_mem - start_mem) / 1024**2:.2f} MB")
except RuntimeError as e:
print(f" 错误: {e}")
# 梯度检查点模式
print("[TEST] 梯度检查点模式...")
try:
torch.cuda.empty_cache()
start_mem = torch.cuda.memory_allocated(device) if torch.cuda.is_available() else 0
model_with_checkpoint.train()
output2 = model_with_checkpoint(x)
loss2 = output2.sum()
loss2.backward()
end_mem = torch.cuda.memory_allocated(device) if torch.cuda.is_available() else 0
print(f" 显存占用: {(end_mem - start_mem) / 1024**2:.2f} MB")
except RuntimeError as e:
print(f" 错误: {e}")
def main():
parser = argparse.ArgumentParser(description='多模态对齐与内存优化')
parser.add_argument('--mode', type=str, default='alignment',
choices=['alignment', 'memory', 'full'],
help='运行模式')
parser.add_argument('--batch_size', type=int, default=4,
help='批次大小')
parser.add_argument('--num_samples', type=int, default=50,
help='样本数量')
args = parser.parse_args()
print(f"[INIT] 初始化多模态对齐系统")
print(f"[CONFIG] 批次大小: {args.batch_size}, 样本数: {args.num_samples}")
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f"[INFO] 使用设备: {device}")
# 初始化对齐模块
aligner = OptimalTransportAlignment(d_in=256, d_out=256, reg=0.1, max_iter=50).to(device)
if args.mode in ['alignment', 'full']:
print("\n" + "="*60)
print("最优传输对齐演示")
print("="*60)
# 创建模拟数据
dataset = MultimodalDataset(num_samples=args.num_samples, cache_features=True)
# 使用长度感知采样器
lengths = dataset.audio_lengths.tolist()
sampler = LengthAwareSampler(lengths, batch_size=args.batch_size, shuffle=True)
dataloader = DataLoader(dataset, batch_sampler=sampler)
all_features = {
'text_features': [],
'audio_aligned': [],
'visual_aligned': [],
'fused_features': [],
'labels': []
}
# 对齐演示
for batch_idx, batch in enumerate(dataloader):
if batch_idx >= 3: # 只演示3个批次
break
text = batch['text'].to(device)
audio = batch['audio'].to(device)
visual = batch['visual'].to(device)
print(f"\n[BATCH {batch_idx+1}] 样本数: {len(text)}")
print(f" 文本特征: {text.shape}")
print(f" 音频特征: {audio.shape}")
print(f" 视觉特征: {visual.shape}")
# 执行对齐
with torch.no_grad():
result = aligner(text, audio, visual, return_coupling=True)
print(f" 融合特征: {result['fused_features'].shape}")
# 保存用于可视化
all_features['text_features'].append(result['text_features'])
all_features['audio_aligned'].append(result['audio_aligned'])
all_features['visual_aligned'].append(result['visual_aligned'])
all_features['fused_features'].append(result['fused_features'])
all_features['labels'].append(batch['label'].numpy())
# 可视化传输计划（仅第一批的第一个样本）
if batch_idx == 0:
if 'coupling_ta' in result:
visualize_optimal_transport(
result['coupling_ta'][0],
title="Text-to-Audio Optimal Transport Coupling"
)
# 合并所有特征
for key in ['text_features', 'audio_aligned', 'visual_aligned', 'fused_features']:
all_features[key] = torch.cat(all_features[key], dim=0)
all_features['labels'] = np.concatenate(all_features['labels'])
print("\n" + "="*60)
print("生成多模态特征对齐可视化")
print("="*60)
visualize_modality_alignment(all_features, all_features['labels'], method='tsne')
if args.mode in ['memory', 'full']:
demonstrate_memory_optimization()
print("\n[COMPLETE] 多模态系统处理完成")
if __name__ == "__main__":
main()
391
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
