*** Received signal 11 *** subprocess.CalledProcessError Segmentation fault

跑pytorch DDP时遇到下列错误：

*** Received signal 11 ***
*** BEGIN MANGLED STACK TRACE ***
/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x76833b)[0x7f00f344333b]
/lib64/libpthread.so.0(+0xf5f0)[0x7f01288ae5f0]
/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/site-packages/apex_C.cpython-36m-x86_64-linux-gnu.so(+0x107cc)[0x7f00ee33a7cc]
/home/sanshuai.lj/software/anaconda3/envs/vilbert/bin/python(_PyCFunction_FastCallDict+0x154)[0x7f0128df0744]
/home/sanshuai.lj/software/anaconda3/envs/vilbert/bin/python(+0x19842c)[0x7f0128e7742c]
......
*** End stack trace ***
Traceback (most recent call last):
  File "/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/site-packages/torch/distributed/launch.py", line 235, in <module>
    main()
  File "/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/site-packages/torch/distributed/launch.py", line 231, in main
    cmd=process.args)
subprocess.CalledProcessError: Command '['/home/sanshuai.lj/software/anaconda3/envs/vilbert/bin/python', '-u', 'train_concap.py', '--local_rank=0', '--from_pretrained', '/mnt/xuesheng_1/bert-base-uncased', '--bert_model', '/mnt/xuesheng_1/bert-base-uncased', '--config_file', 'config/bert_base_6layer_6conect.json', '--learning_rate', '1e-4', '--train_batch_size', '64', '--save_name', 'pretrained', '--distributed']' died with <Signals.SIGABRT: 6>.

换用单卡train时遇到下列错误：

*** Received signal 11 ***
*** BEGIN MANGLED STACK TRACE ***
/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x76833b)[0x7f4560e8333b]
/lib64/libpthread.so.0(+0xf5f0)[0x7f45eb3105f0]
/home/sanshuai.lj/software/anaconda3/envs/vilbert/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so(+0x172cc)[0x7d44dc7932cc]
python(_PyCFunction_FastCallDict+0x154)[0x7f45eb852744]
python(+0x19842c)[0x7f45eb8d942c]
......
*** End stack trace ***
Aborted

pip uninstall tensorflow后单卡train遇到下列错误：

Segmentation fault

pip uninstall apex后单卡train成功。

应该是from apex.normalization.fused_layer_norm import FusedLayerNorm as BertLayerNorm模块出了问题，将其换成手写的pytorch BertLayerNorm模块就可以正常DDP训练了。