fastText文本分类cheatsheet

最新推荐文章于 2024-11-20 16:23:20 发布

原创最新推荐文章于 2024-11-20 16:23:20 发布 · 363 阅读

CC 4.0 BY-SA版权

48 篇文章

订阅专栏

本文详细介绍FastText工具包的使用方法，包括词向量学习、文本分类、句子向量获取及模型量化等关键技术。同时，文章还介绍了如何进行超参数优化，并提供了一系列实用的命令行操作示例。

In order to learn word vectors do:

$ ./fasttext skipgram -input data.txt -output model

Print word vectors for a text file queries.txt containing words.

$ ./fasttext print-word-vectors model.bin < queries.txt

In order to train a text classifier do:

$ ./fasttext supervised -input train.txt -output model

Once the model was trained, you can evaluate it by computing the precision and recall at k (P@k and R@k) on a test set using:

$ ./fasttext test model.bin test.txt 1

In order to obtain the k most likely labels for a piece of text, use:

$ ./fasttext predict model.bin test.txt k

In order to obtain the k most likely labels and their associated probabilities for a piece of text, use:

$ ./fasttext predict-prob model.bin test.txt k

If you want to compute vector representations of sentences or paragraphs, please use:

$ ./fasttext print-sentence-vectors model.bin < text.txt

In order to create a .ftz file with a smaller memory footprint do:

$ ./fasttext quantize -output model

All other commands such as test also work with this model

$ ./fasttext test model.ftz test.txt

Activate hyperparameter optimization with -autotune-validation argument:

$ ./fasttext supervised -input train.txt -output model -autotune-validation valid.txt

Set timeout (in seconds):

$ ./fasttext supervised -input train.txt -output model -autotune-validation valid.txt -autotune-duration 600

Constrain the final model size:

$ ./fasttext supervised -input train.txt -output model -autotune-validation valid.txt -autotune-modelsize 2M