基于mindspore实现知识图谱增强多模态类比推理项目 遇到的mindspore.ops.matmul函数问题

问题描述：

系统环境

硬件环境(Ascend/GPU/CPU): CPU

MindSpore版本: mindspore =2.2.0

Python版本: Python=3.9

操作系统平台: openEuler

问题描述

复现基于mindspore实现知识图谱增强多模态类比推理项目，使用的是github的源码，由于版本问题mindnlp具体版本不知，因此对模型进行了部分更改，具体更新了Embedding层，以及MLM预测头。模型能够求出第一次的loss值，但在进行第二次传播时，模型报错。具体是在进行求BERT层Attention_score出错。为什么输入的张量发生了变化？而且为什么第一次不会报错？

报错信息

key: (16, 12, 96, 64)
value: (16, 12, 96, 64)
query: (16, 12, 96, 64)
key_layer_transposed shape: (16, 12, 64, 96)#输入的张量
value_layer shape before matmul: (16, 12, 96, 64)#输入的张量
Traceback (most recent call last):
File "/home/minda/mind-analogy/mkg_analogy/ms_main.py", line 205, in
main()
File "/home/minda/mind-analogy/mkg_analogy/ms_main.py", line 197, in main
model.train(actual_epoch_num, dataset['train'], callbacks=callback)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/train/model.py", line 1068, in train
self._train(epoch,
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/train/model.py", line 114, in wrapper
func(self, *args, **kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/train/model.py", line 617, in _train
self._train_process(epoch, train_dataset, list_callback, cb_params, initial_epoch, valid_infos)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/train/model.py", line 919, in _train_process outputs = self._train_network(*next_element) File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call raise err File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call output = self._run_construct(args, kwargs) File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct output = self.construct(*cast_inputs, **kwargs) File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/wrap/cell_wrapper.py", line 418, in construct return self._no_sens_impl(*inputs) File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/wrap/cell_wrapper.py", line 433, in _no_sens_impl loss = self.network(*inputs) File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call raise err File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call output = self._run_construct(args, kwargs) File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct output = self.construct(*cast_inputs, **kwargs) File "/home/minda/mind-analogy/mkg_analogy/src/models/model.py", line 50, in construct model_output = self.model(input_ids=input_ids,
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/minda/mind-analogy/mkg_analogy/src/models/modeling_unimo.py", line 841, in construct
outputs = self.unimo(
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/minda/mind-analogy/mkg_analogy/src/models/modeling_unimo.py", line 732, in construct
encoder_outputs = self.encoder(
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/minda/mind-analogy/mkg_analogy/src/models/modeling_unimo.py", line 624, in construct
text_layer_output = text_layer_module(
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/minda/mind-analogy/mkg_analogy/src/models/modeling_unimo.py", line 539, in construct
self_attention_outputs, fusion_output, qks = self.attention(
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/minda/mind-analogy/mkg_analogy/src/models/modeling_unimo.py", line 432, in construct
self_outputs, fusion_output, qks = self.self(
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/minda/mind-analogy/mkg_analogy/src/models/modeling_unimo.py", line 336, in construct
attention_scores = ops.matmul(query_layer, key_layer_transposed) File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindnlp/injection.py", line 70, in wrapper
result = func(*args, **new_kwargs)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/ops/function/math_func.py", line 9690, in matmul
input = _broadcast_to(input, shape1_aligned[:-2], shape_backbone, ndim_aligned)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/ops/function/math_func.py", line 9597, in _broadcast_to F.stop_gradient(size)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/ops/function/grad/grad_func.py", line 1402, in stop_gradient return P.StopGradient()(value)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/ops/primitive.py", line 314, in call return _run_op(self, self.name, args)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/ops/primitive.py", line 913, in _run_op stub = _pynative_executor.run_op_async(obj, op_name, args)
File "/home/minda/anaconda3/envs/mindenv/lib/python3.9/site-packages/mindspore/common/api.py", line 1186, in run_op_async return self._executor.run_op_async(*args)
ValueError: For 'MatMul' the input dimensions must be equal, but got 'x1_col': 768 and 'x2_row': 3072.

BertSelfAttention层代码

def construct(
    self,
    hidden_states,
    attention_mask=None,
    head_mask=None,
    output_attentions=False,
    visual_hidden_state=None,
    output_qks=None,
    sep_idx=None
):
    mixed_query_layer = self.query(hidden_states)
    print(hidden_states.shape)


    key_layer = self.transpose_for_scores(self.key(hidden_states))
    value_layer = self.transpose_for_scores(self.value(hidden_states))
    query_layer = self.transpose_for_scores(mixed_query_layer)
    print("key:",key_layer.shape)
    print("value:",value_layer.shape)
    print("query:",query_layer.shape)

    #qks = (key_layer, value_layer) if output_qks else None
    qks = None
    key_layer_transposed = key_layer.swapaxes(-1,-2)

    print("key_layer_transposed shape:", key_layer_transposed.shape)  # 预期: (64, 12, 64, 96)


    assert query_layer.shape[-1] == key_layer_transposed.shape[-2], \
f"维度不匹配: query_col={query_layer.shape[-1]}, key_row={key_layer_transposed.shape[-2]}"
    print("value_layer shape before matmul:", value_layer.shape)  # 预期: (64, 12, 96, 64)

    # Take the dot product between "query" and "key" to get the raw attention scores.
    attention_scores = ops.matmul(query_layer, key_layer_transposed)  #**代码报错地点**
    print("attention_scores shape:", attention_scores.shape)
    attention_scores = attention_scores / math.sqrt(self.attention_head_size)

解决方法：

可以开启下动态图的同步模式然后在调试代码，现在mindspore默认动态图异步模式运行，可能报错堆栈定位信息不准：

ms.set_context(mode=1, pynative_synchronize=True)

在动态图模式下，你也可以在可能报错的地方通过print的方式打印一下shape等信息，来确认数据是否正确，比如在attention_scores = ops.matmul(query_layer, key_layer_transposed)这一行代码前面打印：

print(query_layer.shape)
print(key_layer_transposed.shape)