SimAlign使用踩坑记录

最近在做无监督词对齐的相关任务，用到了一个开源代码SimAlign
SimAlign是一个通过无监督的方式进行词对齐的工具，其具体的实现方式在它的论文中有具体介绍。

按照官网给出的建议配置环境（Python 3.7, Transformers 3.1.0, Torch 1.5.0. Networkx 2.4）后，安装SimAlign：

pip install --upgrade git+https://github.com/cisnlp/simalign.git#egg=simalign

运行示例代码：

from simalign import SentenceAligner

# making an instance of our model.
# You can specify the embedding model and all alignment settings in the constructor.
myaligner = SentenceAligner(model="bert", token_type="bpe", matching_methods="mai")

# The source and target sentences should be tokenized to words.
src_sentence = ["This", "is", "a", "test", "."]
trg_sentence = ["Das", "ist", "ein", "Test", "."]

# The output is a dictionary with different matching methods.
# Each method has a list of pairs indicating the indexes of aligned words (The alignments are zero-indexed).
alignments = myaligner.get_word_aligns(src_sentence, trg_sentence)

for matching_method in alignments:
    print(matching_method, ":", alignments[matching_method])

# Expected output:
# mwmf (Match): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]
# inter (ArgMax): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]
# itermax (IterMax): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]

本应得到上述输出，但是报错：

Traceback (most recent call last):
  File "align.py", line 13, in <module>
    alignments = myaligner.get_word_aligns(src_sentence, trg_sentence)
  File "/home/users/miniconda3/envs/simalign/lib/python3.7/site-packages/simalign/simalign.py", line 209, in get_word_aligns
    vectors = self.embed_loader.get_embed_list([src_sent, trg_sent]).cpu().detach().numpy()
  File "/home/users/miniconda3/envs/simalign/lib/python3.7/site-packages/simalign/simalign.py", line 62, in get_embed_list
    inputs = self.tokenizer(sent_batch, is_split_into_words=True, padding=True, truncation=True, return_tensors="pt")
  File "/home/users/miniconda3/envs/simalign/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1954, in __call__
    **kwargs,
  File "/home/users/miniconda3/envs/simalign/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2139, in batch_encode_plus
    **kwargs,
  File "/home/users/miniconda3/envs/simalign/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 531, in _batch_encode_plus
    ids, pair_ids = ids_or_pair_ids
ValueError: too many values to unpack (expected 2)

这个情况是由于transformers的版本造成的，卸载transformers，安装最新版本的transformers（我安装的是4.16.2）就解决了。

pip uninstall transformers
pip install transformers

得到的输出如下：