BPE(Byte-Pair Encoding )代码实现

原创

已于 2024-01-11 14:46:36 修改 · 884 阅读

9 ·

CC 4.0 BY-SA版权

文章标签：

#机器学习 #人工智能 #数据挖掘 #算法 #python #nlp

于 2024-01-11 12:26:01 首次发布

BPE 是使用最广泛的sub-word tokenization算法之一。尽管贪婪，但它具有良好的性能,并被作为机器翻译等主流NLP任务的首选tokenize方法之一。

BPE算法原理传送门

1. Byte-Pair Encoding Tokenizer Training


import pandas as pd

# Import gc, a library for controlling the garbage collector
import gc

# Import various classes and functions from the tokenizers library, which is used for creating and using custom tokenizers 
from tokenizers import (
    decoders,
    models,
    normalizers,
    pre_tokenizers,
    processors,
    trainers,
    Tokenizer,
)

# Import PreTrainedTokenizerFast, a class for using fast tokenizers from the transformers library
from transformers import PreTrainedTokenizerFast

# Import TfidfVectorizer, a class for transforming text into TF-IDF features
from sklearn.feature_extraction.text import TfidfVectorizer

# Import tqdm, a library for displaying progress bars 
from tqdm.auto import tqdm

# Import Dataset, a class for working with datasets in a standardized way 
from datasets import Dataset

# Set the LOWERCASE flag to False
LOWERCASE = False 

# Set the VOCAB_SIZE to 10000000.
# This means that the maximum number of words in the vocabulary will be 10 million.
VOCAB_SIZE = 10000000