Kaldi Debug:ERROR (ivector-extractor-sum-accs[5.2]:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1]
最近在跑Kaldi提iVector做说话人识别的程序,在训练T矩阵的时候遇到了以下问题,困扰了我好几天,网上对这个问题的资料不是很多,在这里记录一下。
错误出现在运行sid/train_ivector_extractor.sh
中的
Args[$j]=`echo "ivector-extractor-acc-stats --num-threads=$num_threads --num-samples-for-weights=$num_samples_for_weights $dir/$x.ie '$feats' 'ark,s,cs:gunzip -c $dir/post.JOB.gz|' -|" | sed s/JOB/$j/g`
报错的信息保存在exp\extractor\log\acc0.4.log中
# ivector-extractor-sum-accs --parallel=true "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.13.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.14.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.15.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.16.gz|' -|" exp/extractor/acc.0.4
# Started at Fri Sep 20 22:55:54 CST 2019
#
ivector-extractor-sum-accs --parallel=true "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.13.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.14.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.15.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.16.gz|' -|" exp/extractor/acc.0.4
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.15.gz|' -
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.14.gz|' -
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.16.gz|' -
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.13.gz|' -
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:-
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:-
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:-
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:-
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:-
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:-
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:-
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:-
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:-
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:-
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:-
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:-
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
ERROR (ivector-extractor-sum-accs[5.2]:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>
[ Stack-Trace: ]
kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*)
kaldi::MessageLogger::~MessageLogger()
kaldi::ExpectToken(std::istream&, bool, char const*)
kaldi::IvectorExtractorStats::Read(std::istream&, bool, bool)
main
__libc_start_main
ivector-extractor-sum-accs() [0x40c649]
# Accounting: time=633 threads=1
# Ended (code 255) at Fri Sep 20 23:06:27 CST 2019, elapsed time 633 seconds
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
WARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe gunzip -c exp/extractor/post.14.gz| had nonzero return status 13
WARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- | had nonzero return status 36096
std::bad_allocWARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe gunzip -c exp/extractor/post.15.gz| had nonzero return status 13
WARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- | had nonzero return status 36096
std::bad_alloc
最主要的就是ERROR (ivector-extractor-sum-accs[5.2]:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>
这句。
这个错误是由并行任务开的太多而内存不够大造成的,解决办法就是减小sid/train_ivector_extractor.sh
脚本中设置的并行任务数nj。
# Begin configuration section.
nj=10 # this is the number of separate queue jobs we run, but each one
# contains num_processes sub-jobs.. the real number of threads we
# run is nj * num_processes * num_threads, and the number of
# separate pieces of data is nj * num_processes.
num_threads=4
num_processes=4 # each job runs this many processes, each with --num-threads threads
cmd="run.pl"
脚本默认nj=10,最后会产生4410=80个线程,并把训练数据拆分成4*10=40个文件。经过我的测试,训练数据集使用Voxceleb1和Voxceleb2-dev集,nj=2时,会占用大约32G的内存,可以根据自己的内存和训练数据集的大小修改下脚本。
之前在另一台服务器上跑的时候,可能也是因为线程数设置的太大出了错,错误都是来自于同一个脚本的同一个位置,但是报的错误内容不一样:
ERROR (ivector-extractor-acc-stats[5.2]:ReadBasicType<float>():io-funcs.cc:85) ReadBasicType: expected float, saw -1, at file position -
……
WARNING (ivector-extractor-acc-stats[5.2]:Read():posterior.cc:150) Exception caught reading table of posteriors.
WARNING (ivector-extractor-acc-stats[5.2]:ReadNextObject():util/kaldi-table-inl.h:1973) Object read failed, reading archive 'gunzip -c exp/extractor/post.8.gz|'
WARNING (ivector-extractor-acc-stats[5.2]:main():ivector-extractor-acc-stats.cc:127) No posteriors for utterance id02087-8uXHWdo0cYw-00020
WARNING (ivector-extractor-acc-stats[5.2]:main():ivector-extractor-acc-stats.cc:127) No posteriors for utterance id02087-8uXHWdo0cYw-00022
应该可以用同样的方法来解决。
参考自Dan大神在论坛里的回复
https://groups.google.com/forum/#!msg/kaldi-help/D1rU4EaTmjw/AnCis3BPAgAJ