kaldi debug：Failed to read token [started at file position -1], expected <IvectorExtractorStats>_failed to read token[started at file position -1]-优快云博客

本文链接：https://blog.youkuaiyun.com/qijiajun97/article/details/101149845

本文详细解析了在Kaldi中进行说话人识别时遇到的iVector提取错误，特别是“Failed to read token”问题，揭示了内存不足是根本原因，并提供了减少并行任务数的解决方案。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Kaldi Debug:ERROR (ivector-extractor-sum-accs[5.2]:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1]

最近在跑Kaldi提iVector做说话人识别的程序，在训练T矩阵的时候遇到了以下问题，困扰了我好几天，网上对这个问题的资料不是很多，在这里记录一下。
错误出现在运行sid/train_ivector_extractor.sh中的

Args[$j]=`echo "ivector-extractor-acc-stats --num-threads=$num_threads --num-samples-for-weights=$num_samples_for_weights $dir/$x.ie '$feats' 'ark,s,cs:gunzip -c $dir/post.JOB.gz|' -|" | sed s/JOB/$j/g`

报错的信息保存在exp\extractor\log\acc0.4.log中

# ivector-extractor-sum-accs --parallel=true "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.13.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.14.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.15.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.16.gz|' -|" exp/extractor/acc.0.4 
# Started at Fri Sep 20 22:55:54 CST 2019
#
ivector-extractor-sum-accs --parallel=true "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.13.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.14.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.15.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.16.gz|' -|" exp/extractor/acc.0.4 
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.15.gz|' - 
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.14.gz|' - 
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.16.gz|' - 
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor/0.ie 'ark,s,cs:add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor/post.13.gz|' - 
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- 
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- 
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/13/vad.scp ark:- 
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- 
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- 
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/13/feats.scp ark:- 
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- 
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- 
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- 
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- 
select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/16/vad.scp ark:- 
add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/16/feats.scp ark:- 
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
ERROR (ivector-extractor-sum-accs[5.2]:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>

[ Stack-Trace: ]

kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*)
kaldi::MessageLogger::~MessageLogger()
kaldi::ExpectToken(std::istream&, bool, char const*)
kaldi::IvectorExtractorStats::Read(std::istream&, bool, bool)
main
__libc_start_main
ivector-extractor-sum-accs() [0x40c649]


# Accounting: time=633 threads=1
# Ended (code 255) at Fri Sep 20 23:06:27 CST 2019, elapsed time 633 seconds
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (ivector-extractor-acc-stats[5.2]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
WARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe gunzip -c exp/extractor/post.14.gz| had nonzero return status 13
WARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/14/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/14/vad.scp ark:- | had nonzero return status 36096
std::bad_allocWARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe gunzip -c exp/extractor/post.15.gz| had nonzero return status 13
WARNING (ivector-extractor-acc-stats[5.2]:Close():kaldi-io.cc:501) Pipe add-deltas --delta-window=3 --delta-order=2 scp:data/train_100k/split20/15/feats.scp ark:- | apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train_100k/split20/15/vad.scp ark:- | had nonzero return status 36096
std::bad_alloc

最主要的就是ERROR (ivector-extractor-sum-accs[5.2]:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>这句。
这个错误是由并行任务开的太多而内存不够大造成的，解决办法就是减小sid/train_ivector_extractor.sh脚本中设置的并行任务数nj。

# Begin configuration section.
nj=10   # this is the number of separate queue jobs we run, but each one
        # contains num_processes sub-jobs.. the real number of threads we
        # run is nj * num_processes * num_threads, and the number of
        # separate pieces of data is nj * num_processes.
num_threads=4
num_processes=4 # each job runs this many processes, each with --num-threads threads
cmd="run.pl"

脚本默认nj=10，最后会产生4410=80个线程，并把训练数据拆分成4*10=40个文件。经过我的测试，训练数据集使用Voxceleb1和Voxceleb2-dev集，nj=2时，会占用大约32G的内存，可以根据自己的内存和训练数据集的大小修改下脚本。

之前在另一台服务器上跑的时候，可能也是因为线程数设置的太大出了错，错误都是来自于同一个脚本的同一个位置，但是报的错误内容不一样：

ERROR (ivector-extractor-acc-stats[5.2]:ReadBasicType<float>():io-funcs.cc:85) ReadBasicType: expected float, saw -1, at file position -
……
WARNING (ivector-extractor-acc-stats[5.2]:Read():posterior.cc:150) Exception caught reading table of posteriors. 
WARNING (ivector-extractor-acc-stats[5.2]:ReadNextObject():util/kaldi-table-inl.h:1973) Object read failed, reading archive 'gunzip -c exp/extractor/post.8.gz|'
WARNING (ivector-extractor-acc-stats[5.2]:main():ivector-extractor-acc-stats.cc:127) No posteriors for utterance id02087-8uXHWdo0cYw-00020
WARNING (ivector-extractor-acc-stats[5.2]:main():ivector-extractor-acc-stats.cc:127) No posteriors for utterance id02087-8uXHWdo0cYw-00022

应该可以用同样的方法来解决。

参考自Dan大神在论坛里的回复
https://groups.google.com/forum/#!msg/kaldi-help/D1rU4EaTmjw/AnCis3BPAgAJ