使用kashgari快速搭建文本分类模型，对比各个模型间的正确率。
import kashgari
from kashgari.embeddings import BERTEmbedding
bert_embed = BERTEmbedding('chinese_roberta_wwm_large', task=kashgari.CLASSIFICATION, sequence_length=256)
导入数据集
from kashgari.corpus import SMP2018ECDTCorpus
train_x, train_y = SMP2018ECDTCorpus.load_data('train')
valid_x, valid_y = SMP2018ECDTCorpus.load_data('valid')
test_x, test_y = SMP2018ECDTCorpus.load_data('test')
train_x[0:5]
[['中', '央', '十', '一', '台'],
['沙', '眼', '。'],
['我', '想', '知', '道', '科', '大', '讯', '飞', '的', '市', '值'],
['打', '开', 's', 'k', 'y', '电', '话'],
['上', '饶', '到', '南', '昌', '的', '火', '车', '票']]
train_y[0:5]
['tvchannel', 'health', 'stock', 'app', 'train']
set(train_y)
{'app',
'bus',
'calc',
'chat',
'cinemas',
'contacts',
'cookbook',
'datetime',
'email',
'epg',
'flight',
'health',
'lottery',
'map',
'match',
'message',
'music',
'news',
'novel',
'poetry',
'radio',
'riddle',
'schedule',
'stock',
'telephone',
'train',
'translation',
'tvchannel',
'video',
'weather',
'website'}
len(set(train_y))
31
本数据集共31个标签，是31分类任务。
Bert_BiLSTM_Model
from kashgari.tasks.classification import BiLSTM_Model
model_1 = BiLSTM_Model(bert_embed)
model_1.fit(train_x, train_y, valid_x, valid_y,epochs=2, batch_size=64)
result_1 = model_1.evaluate(test_x, test_y)
model_1.save('model/BERT_BiLSTM.h5')
Model: "model_4"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
Input-Token (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Input-Segment (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Embedding-Token (TokenEmbedding [(None, 128, 1024), 21635072 Input-Token[0][0]
__________________________________________________________________________________________________
Embedding-Segment (Embedding) (None, 128, 1024) 2048 Input-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Token-Segment (Add) (None, 128, 1024) 0 Embedding-Token[0][0]
Embedding-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Position (PositionEmb (None, 128, 1024) 131072 Embedding-Token-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Dropout (Dropout) (None, 128, 1024) 0 Embedding-Position[0][0]
__________________________________________________________________________________________________
Embedding-Norm (LayerNormalizat (None, 128, 1024) 2048 Embedding-Dropout[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Embedding-Norm[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Embedding-Norm[0][0]
Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-1-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-1-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention-
Encoder-1-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-1-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-1-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-1-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-FeedForward-Norm[0][0]
Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-2-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-2-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention-
Encoder-2-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-2-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-2-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-2-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-FeedForward-Norm[0][0]
Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-3-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-3-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention-
Encoder-3-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-3-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-3-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-3-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-FeedForward-Norm[0][0]
Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-4-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-4-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention-
Encoder-4-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-4-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-4-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-4-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-FeedForward-Norm[0][0]
Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-5-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-5-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention-
Encoder-5-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-5-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-5-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-5-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-FeedForward-Norm[0][0]
Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-6-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-6-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention-
Encoder-6-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-6-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-6-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-6-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-FeedForward-Norm[0][0]
Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-7-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-7-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention-
Encoder-7-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-7-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-7-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-7-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-FeedForward-Norm[0][0]
Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-8-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-8-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention-
Encoder-8-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-8-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-8-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-8-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-FeedForward-Norm[0][0]
Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-9-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-9-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention-
Encoder-9-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-9-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-9-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-9-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-9-FeedForward-Norm[0][0]
Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward-Dropout (None, 128, 1024) 0 Encoder-10-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-10-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-10-MultiHeadSelfAttention
Encoder-10-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-10-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-10-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-10-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-10-FeedForward-Norm[0][0]
Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward-Dropout (None, 128, 1024) 0 Encoder-11-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-11-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-11-MultiHeadSelfAttention
Encoder-11-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-11-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-11-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-11-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-11-FeedForward-Norm[0][0]
Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward-Dropout (None, 128, 1024) 0 Encoder-12-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-12-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-12-MultiHeadSelfAttention
Encoder-12-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-12-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-12-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-12-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-12-FeedForward-Norm[0][0]
Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward-Dropout (None, 128, 1024) 0 Encoder-13-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-13-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-13-MultiHeadSelfAttention
Encoder-13-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-13-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-13-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-13-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-13-FeedForward-Norm[0][0]
Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward-Dropout (None, 128, 1024) 0 Encoder-14-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-14-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-14-MultiHeadSelfAttention
Encoder-14-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-14-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-14-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-14-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-14-FeedForward-Norm[0][0]
Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward-Dropout (None, 128, 1024) 0 Encoder-15-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-15-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-15-MultiHeadSelfAttention
Encoder-15-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-15-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-15-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-15-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-15-FeedForward-Norm[0][0]
Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward-Dropout (None, 128, 1024) 0 Encoder-16-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-16-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-16-MultiHeadSelfAttention
Encoder-16-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-16-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-16-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-16-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-16-FeedForward-Norm[0][0]
Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward-Dropout (None, 128, 1024) 0 Encoder-17-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-17-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-17-MultiHeadSelfAttention
Encoder-17-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-17-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-17-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-17-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-17-FeedForward-Norm[0][0]
Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward-Dropout (None, 128, 1024) 0 Encoder-18-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-18-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-18-MultiHeadSelfAttention
Encoder-18-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-18-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-18-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-18-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-18-FeedForward-Norm[0][0]
Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward-Dropout (None, 128, 1024) 0 Encoder-19-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-19-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-19-MultiHeadSelfAttention
Encoder-19-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-19-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-19-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-19-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-19-FeedForward-Norm[0][0]
Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward-Dropout (None, 128, 1024) 0 Encoder-20-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-20-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-20-MultiHeadSelfAttention
Encoder-20-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-20-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-20-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-20-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-20-FeedForward-Norm[0][0]
Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward-Dropout (None, 128, 1024) 0 Encoder-21-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-21-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-21-MultiHeadSelfAttention
Encoder-21-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-21-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-21-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-21-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-21-FeedForward-Norm[0][0]
Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward-Dropout (None, 128, 1024) 0 Encoder-22-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-22-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-22-MultiHeadSelfAttention
Encoder-22-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-22-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-22-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-22-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-22-FeedForward-Norm[0][0]
Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward-Dropout (None, 128, 1024) 0 Encoder-23-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-23-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-23-MultiHeadSelfAttention
Encoder-23-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-23-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-23-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-23-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-23-FeedForward-Norm[0][0]
Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward-Dropout (None, 128, 1024) 0 Encoder-24-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-24-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-24-MultiHeadSelfAttention
Encoder-24-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-24-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-24-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-Output (Concatenate) (None, 128, 4096) 0 Encoder-21-FeedForward-Norm[0][0]
Encoder-22-FeedForward-Norm[0][0]
Encoder-23-FeedForward-Norm[0][0]
Encoder-24-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
non_masking_layer (NonMaskingLa (None, 128, 4096) 0 Encoder-Output[0][0]
__________________________________________________________________________________________________
bidirectional (Bidirectional) (None, 256) 4326400 non_masking_layer[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 31) 7967 bidirectional[0][0]
==================================================================================================
Total params: 328,413,983
Trainable params: 4,334,367
Non-trainable params: 324,079,616
__________________________________________________________________________________________________
Epoch 1/2
30/30 [==============================] - 413s 14s/step - loss: 1.2704 - acc: 0.6885 - val_loss: 0.5361 - val_acc: 0.8517
Epoch 2/2
30/30 [==============================] - 410s 14s/step - loss: 0.3376 - acc: 0.9261 - val_loss: 0.3924 - val_acc: 0.9019
precision recall f1-score support
app 1.0000 0.4444 0.6154 18
bus 1.0000 0.5000 0.6667 8
calc 1.0000 1.0000 1.0000 8
chat 0.9490 0.9675 0.9582 154
cinemas 0.6000 0.3750 0.4615 8
contacts 1.0000 0.9000 0.9474 10
cookbook 1.0000 1.0000 1.0000 89
datetime 0.7500 1.0000 0.8571 6
email 1.0000 0.7500 0.8571 8
epg 0.8333 0.8333 0.8333 36
flight 0.9524 0.9524 0.9524 21
health 1.0000 1.0000 1.0000 19
lottery 1.0000 1.0000 1.0000 8
map 0.9130 0.9130 0.9130 23
match 0.8889 1.0000 0.9412 8
message 0.9545 1.0000 0.9767 21
music 0.9130 0.9545 0.9333 22
news 1.0000 0.8000 0.8889 20
novel 0.7778 0.8750 0.8235 8
poetry 0.9189 1.0000 0.9577 34
radio 1.0000 0.7500 0.8571 8
riddle 1.0000 1.0000 1.0000 11
schedule 1.0000 0.7778 0.8750 9
stock 1.0000 0.9583 0.9787 24
telephone 1.0000 1.0000 1.0000 21
train 0.8000 1.0000 0.8889 24
translation 1.0000 0.9524 0.9756 21
tvchannel 0.7895 0.6522 0.7143 23
video 0.8226 0.8500 0.8361 60
weather 0.9545 0.9545 0.9545 22
website 0.5312 0.9444 0.6800 18
accuracy 0.9130 770
macro avg 0.9145 0.8744 0.8821 770
weighted avg 0.9226 0.9130 0.9111 770
Bert_BiGRU_Model
from kashgari.tasks.classification import BiGRU_Model
model_2 = BiGRU_Model(bert_embed)
model_2.fit(train_x, train_y, valid_x, valid_y,epochs=2, batch_size=64)
result_2 = model_2.evaluate(test_x, test_y)
model_2.save('model/BERT_BiGRU.h5')
Model: "model_5"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
Input-Token (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Input-Segment (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Embedding-Token (TokenEmbedding [(None, 128, 1024), 21635072 Input-Token[0][0]
__________________________________________________________________________________________________
Embedding-Segment (Embedding) (None, 128, 1024) 2048 Input-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Token-Segment (Add) (None, 128, 1024) 0 Embedding-Token[0][0]
Embedding-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Position (PositionEmb (None, 128, 1024) 131072 Embedding-Token-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Dropout (Dropout) (None, 128, 1024) 0 Embedding-Position[0][0]
__________________________________________________________________________________________________
Embedding-Norm (LayerNormalizat (None, 128, 1024) 2048 Embedding-Dropout[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Embedding-Norm[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Embedding-Norm[0][0]
Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-1-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-1-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention-
Encoder-1-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-1-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-1-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-1-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-FeedForward-Norm[0][0]
Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-2-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-2-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention-
Encoder-2-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-2-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-2-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-2-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-FeedForward-Norm[0][0]
Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-3-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-3-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention-
Encoder-3-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-3-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-3-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-3-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-FeedForward-Norm[0][0]
Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-4-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-4-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention-
Encoder-4-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-4-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-4-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-4-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-FeedForward-Norm[0][0]
Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-5-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-5-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention-
Encoder-5-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-5-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-5-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-5-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-FeedForward-Norm[0][0]
Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-6-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-6-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention-
Encoder-6-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-6-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-6-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-6-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-FeedForward-Norm[0][0]
Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-7-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-7-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention-
Encoder-7-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-7-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-7-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-7-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-FeedForward-Norm[0][0]
Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-8-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-8-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention-
Encoder-8-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-8-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-8-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-8-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-FeedForward-Norm[0][0]
Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-9-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-9-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention-
Encoder-9-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-9-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-9-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-9-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-9-FeedForward-Norm[0][0]
Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward-Dropout (None, 128, 1024) 0 Encoder-10-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-10-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-10-MultiHeadSelfAttention
Encoder-10-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-10-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-10-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-10-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-10-FeedForward-Norm[0][0]
Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward-Dropout (None, 128, 1024) 0 Encoder-11-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-11-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-11-MultiHeadSelfAttention
Encoder-11-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-11-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-11-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-11-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-11-FeedForward-Norm[0][0]
Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward-Dropout (None, 128, 1024) 0 Encoder-12-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-12-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-12-MultiHeadSelfAttention
Encoder-12-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-12-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-12-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-12-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-12-FeedForward-Norm[0][0]
Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward-Dropout (None, 128, 1024) 0 Encoder-13-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-13-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-13-MultiHeadSelfAttention
Encoder-13-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-13-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-13-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-13-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-13-FeedForward-Norm[0][0]
Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward-Dropout (None, 128, 1024) 0 Encoder-14-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-14-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-14-MultiHeadSelfAttention
Encoder-14-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-14-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-14-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-14-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-14-FeedForward-Norm[0][0]
Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward-Dropout (None, 128, 1024) 0 Encoder-15-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-15-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-15-MultiHeadSelfAttention
Encoder-15-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-15-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-15-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-15-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-15-FeedForward-Norm[0][0]
Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward-Dropout (None, 128, 1024) 0 Encoder-16-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-16-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-16-MultiHeadSelfAttention
Encoder-16-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-16-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-16-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-16-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-16-FeedForward-Norm[0][0]
Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward-Dropout (None, 128, 1024) 0 Encoder-17-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-17-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-17-MultiHeadSelfAttention
Encoder-17-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-17-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-17-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-17-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-17-FeedForward-Norm[0][0]
Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward-Dropout (None, 128, 1024) 0 Encoder-18-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-18-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-18-MultiHeadSelfAttention
Encoder-18-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-18-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-18-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-18-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-18-FeedForward-Norm[0][0]
Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward-Dropout (None, 128, 1024) 0 Encoder-19-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-19-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-19-MultiHeadSelfAttention
Encoder-19-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-19-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-19-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-19-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-19-FeedForward-Norm[0][0]
Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward-Dropout (None, 128, 1024) 0 Encoder-20-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-20-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-20-MultiHeadSelfAttention
Encoder-20-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-20-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-20-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-20-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-20-FeedForward-Norm[0][0]
Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward-Dropout (None, 128, 1024) 0 Encoder-21-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-21-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-21-MultiHeadSelfAttention
Encoder-21-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-21-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-21-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-21-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-21-FeedForward-Norm[0][0]
Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward-Dropout (None, 128, 1024) 0 Encoder-22-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-22-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-22-MultiHeadSelfAttention
Encoder-22-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-22-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-22-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-22-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-22-FeedForward-Norm[0][0]
Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward-Dropout (None, 128, 1024) 0 Encoder-23-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-23-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-23-MultiHeadSelfAttention
Encoder-23-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-23-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-23-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-23-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-23-FeedForward-Norm[0][0]
Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward-Dropout (None, 128, 1024) 0 Encoder-24-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-24-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-24-MultiHeadSelfAttention
Encoder-24-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-24-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-24-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-Output (Concatenate) (None, 128, 4096) 0 Encoder-21-FeedForward-Norm[0][0]
Encoder-22-FeedForward-Norm[0][0]
Encoder-23-FeedForward-Norm[0][0]
Encoder-24-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
non_masking_layer (NonMaskingLa (None, 128, 4096) 0 Encoder-Output[0][0]
__________________________________________________________________________________________________
bidirectional_1 (Bidirectional) (None, 256) 3244800 non_masking_layer[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 31) 7967 bidirectional_1[0][0]
==================================================================================================
Total params: 327,332,383
Trainable params: 3,252,767
Non-trainable params: 324,079,616
__________________________________________________________________________________________________
Epoch 1/2
30/30 [==============================] - 413s 14s/step - loss: 1.3147 - acc: 0.6805 - val_loss: 0.4360 - val_acc: 0.8923
Epoch 2/2
30/30 [==============================] - 410s 14s/step - loss: 0.2226 - acc: 0.9452 - val_loss: 0.2943 - val_acc: 0.9282
precision recall f1-score support
app 0.8235 0.7778 0.8000 18
bus 1.0000 0.8750 0.9333 8
calc 1.0000 1.0000 1.0000 8
chat 0.9671 0.9545 0.9608 154
cinemas 0.5714 0.5000 0.5333 8
contacts 0.8333 1.0000 0.9091 10
cookbook 1.0000 1.0000 1.0000 89
datetime 0.7500 1.0000 0.8571 6
email 0.8750 0.8750 0.8750 8
epg 0.8857 0.8611 0.8732 36
flight 0.9545 1.0000 0.9767 21
health 1.0000 1.0000 1.0000 19
lottery 1.0000 1.0000 1.0000 8
map 0.8846 1.0000 0.9388 23
match 0.8000 1.0000 0.8889 8
message 0.9545 1.0000 0.9767 21
music 0.8636 0.8636 0.8636 22
news 0.8636 0.9500 0.9048 20
novel 1.0000 0.8750 0.9333 8
poetry 0.9167 0.9706 0.9429 34
radio 1.0000 0.7500 0.8571 8
riddle 1.0000 1.0000 1.0000 11
schedule 1.0000 0.7778 0.8750 9
stock 1.0000 0.9583 0.9787 24
telephone 1.0000 1.0000 1.0000 21
train 0.9600 1.0000 0.9796 24
translation 0.9091 0.9524 0.9302 21
tvchannel 0.8421 0.6957 0.7619 23
video 0.9153 0.9000 0.9076 60
weather 0.9545 0.9545 0.9545 22
website 0.8333 0.8333 0.8333 18
accuracy 0.9338 770
macro avg 0.9148 0.9137 0.9112 770
weighted avg 0.9347 0.9338 0.9329 770
Bert_CNN_Model
from kashgari.tasks.classification import CNN_Model
model_3 = CNN_Model(bert_embed)
model_3.fit(train_x, train_y, valid_x, valid_y,epochs=2, batch_size=64)
result_3 = model_3.evaluate(test_x, test_y)
model_3.save('model/BERT_CNN.h5')
Model: "model_6"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
Input-Token (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Input-Segment (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Embedding-Token (TokenEmbedding [(None, 128, 1024), 21635072 Input-Token[0][0]
__________________________________________________________________________________________________
Embedding-Segment (Embedding) (None, 128, 1024) 2048 Input-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Token-Segment (Add) (None, 128, 1024) 0 Embedding-Token[0][0]
Embedding-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Position (PositionEmb (None, 128, 1024) 131072 Embedding-Token-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Dropout (Dropout) (None, 128, 1024) 0 Embedding-Position[0][0]
__________________________________________________________________________________________________
Embedding-Norm (LayerNormalizat (None, 128, 1024) 2048 Embedding-Dropout[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Embedding-Norm[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Embedding-Norm[0][0]
Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-1-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-1-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention-
Encoder-1-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-1-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-1-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-1-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-FeedForward-Norm[0][0]
Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-2-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-2-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention-
Encoder-2-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-2-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-2-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-2-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-FeedForward-Norm[0][0]
Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-3-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-3-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention-
Encoder-3-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-3-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-3-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-3-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-FeedForward-Norm[0][0]
Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-4-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-4-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention-
Encoder-4-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-4-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-4-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-4-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-FeedForward-Norm[0][0]
Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-5-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-5-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention-
Encoder-5-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-5-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-5-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-5-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-FeedForward-Norm[0][0]
Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-6-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-6-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention-
Encoder-6-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-6-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-6-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-6-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-FeedForward-Norm[0][0]
Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-7-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-7-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention-
Encoder-7-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-7-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-7-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-7-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-FeedForward-Norm[0][0]
Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-8-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-8-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention-
Encoder-8-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-8-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-8-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-8-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-FeedForward-Norm[0][0]
Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-9-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-9-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention-
Encoder-9-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-9-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-9-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-9-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-9-FeedForward-Norm[0][0]
Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward-Dropout (None, 128, 1024) 0 Encoder-10-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-10-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-10-MultiHeadSelfAttention
Encoder-10-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-10-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-10-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-10-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-10-FeedForward-Norm[0][0]
Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-11-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-11-FeedForward-Dropout (None, 128, 1024) 0 Encoder-11-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-11-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-11-MultiHeadSelfAttention
Encoder-11-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-11-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-11-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-11-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-11-FeedForward-Norm[0][0]
Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-12-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-12-FeedForward-Dropout (None, 128, 1024) 0 Encoder-12-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-12-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-12-MultiHeadSelfAttention
Encoder-12-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-12-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-12-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-12-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-12-FeedForward-Norm[0][0]
Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-13-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-13-FeedForward-Dropout (None, 128, 1024) 0 Encoder-13-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-13-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-13-MultiHeadSelfAttention
Encoder-13-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-13-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-13-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-13-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-13-FeedForward-Norm[0][0]
Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-14-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-14-FeedForward-Dropout (None, 128, 1024) 0 Encoder-14-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-14-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-14-MultiHeadSelfAttention
Encoder-14-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-14-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-14-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-14-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-14-FeedForward-Norm[0][0]
Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-15-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-15-FeedForward-Dropout (None, 128, 1024) 0 Encoder-15-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-15-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-15-MultiHeadSelfAttention
Encoder-15-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-15-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-15-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-15-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-15-FeedForward-Norm[0][0]
Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-16-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-16-FeedForward-Dropout (None, 128, 1024) 0 Encoder-16-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-16-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-16-MultiHeadSelfAttention
Encoder-16-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-16-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-16-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-16-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-16-FeedForward-Norm[0][0]
Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-17-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-17-FeedForward-Dropout (None, 128, 1024) 0 Encoder-17-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-17-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-17-MultiHeadSelfAttention
Encoder-17-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-17-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-17-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-17-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-17-FeedForward-Norm[0][0]
Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-18-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-18-FeedForward-Dropout (None, 128, 1024) 0 Encoder-18-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-18-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-18-MultiHeadSelfAttention
Encoder-18-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-18-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-18-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-18-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-18-FeedForward-Norm[0][0]
Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-19-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-19-FeedForward-Dropout (None, 128, 1024) 0 Encoder-19-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-19-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-19-MultiHeadSelfAttention
Encoder-19-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-19-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-19-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-19-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-19-FeedForward-Norm[0][0]
Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-20-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-20-FeedForward-Dropout (None, 128, 1024) 0 Encoder-20-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-20-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-20-MultiHeadSelfAttention
Encoder-20-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-20-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-20-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-20-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-20-FeedForward-Norm[0][0]
Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-21-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-21-FeedForward-Dropout (None, 128, 1024) 0 Encoder-21-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-21-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-21-MultiHeadSelfAttention
Encoder-21-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-21-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-21-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-21-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-21-FeedForward-Norm[0][0]
Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-22-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-22-FeedForward-Dropout (None, 128, 1024) 0 Encoder-22-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-22-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-22-MultiHeadSelfAttention
Encoder-22-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-22-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-22-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-22-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-22-FeedForward-Norm[0][0]
Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-23-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-23-FeedForward-Dropout (None, 128, 1024) 0 Encoder-23-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-23-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-23-MultiHeadSelfAttention
Encoder-23-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-23-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-23-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-23-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-23-FeedForward-Norm[0][0]
Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-24-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-24-FeedForward-Dropout (None, 128, 1024) 0 Encoder-24-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-24-FeedForward-Add (Add (None, 128, 1024) 0 Encoder-24-MultiHeadSelfAttention
Encoder-24-FeedForward-Dropout[0]
__________________________________________________________________________________________________
Encoder-24-FeedForward-Norm (La (None, 128, 1024) 2048 Encoder-24-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-Output (Concatenate) (None, 128, 4096) 0 Encoder-21-FeedForward-Norm[0][0]
Encoder-22-FeedForward-Norm[0][0]
Encoder-23-FeedForward-Norm[0][0]
Encoder-24-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
non_masking_layer (NonMaskingLa (None, 128, 4096) 0 Encoder-Output[0][0]
__________________________________________________________________________________________________
conv1d (Conv1D) (None, 124, 128) 2621568 non_masking_layer[0][0]
__________________________________________________________________________________________________
global_max_pooling1d (GlobalMax (None, 128) 0 conv1d[0][0]
__________________________________________________________________________________________________
dense_2 (Dense) (None, 64) 8256 global_max_pooling1d[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 31) 2015 dense_2[0][0]
==================================================================================================
Total params: 326,711,455
Trainable params: 2,631,839
Non-trainable params: 324,079,616
__________________________________________________________________________________________________
Epoch 1/2
30/30 [==============================] - 372s 12s/step - loss: 2.0315 - acc: 0.5742 - val_loss: 0.7069 - val_acc: 0.8254
Epoch 2/2
30/30 [==============================] - 369s 12s/step - loss: 0.3203 - acc: 0.9187 - val_loss: 0.4566 - val_acc: 0.8900
precision recall f1-score support
app 0.8182 0.5000 0.6207 18
bus 1.0000 1.0000 1.0000 8
calc 1.0000 0.8750 0.9333 8
chat 0.9551 0.9675 0.9613 154
cinemas 0.5000 0.5000 0.5000 8
contacts 1.0000 1.0000 1.0000 10
cookbook 1.0000 1.0000 1.0000 89
datetime 0.7143 0.8333 0.7692 6
email 0.7500 0.7500 0.7500 8
epg 0.8000 0.8889 0.8421 36
flight 0.9091 0.9524 0.9302 21
health 1.0000 0.9474 0.9730 19
lottery 1.0000 1.0000 1.0000 8
map 0.9565 0.9565 0.9565 23
match 0.7273 1.0000 0.8421 8
message 0.9524 0.9524 0.9524 21
music 0.9286 0.5909 0.7222 22
news 0.9091 1.0000 0.9524 20
novel 1.0000 0.8750 0.9333 8
poetry 0.8500 1.0000 0.9189 34
radio 1.0000 0.6250 0.7692 8
riddle 1.0000 1.0000 1.0000 11
schedule 1.0000 0.7778 0.8750 9
stock 1.0000 0.9583 0.9787 24
telephone 1.0000 1.0000 1.0000 21
train 0.9600 1.0000 0.9796 24
translation 1.0000 0.9524 0.9756 21
tvchannel 0.7391 0.7391 0.7391 23
video 0.8103 0.7833 0.7966 60
weather 0.9524 0.9091 0.9302 22
website 0.6154 0.8889 0.7273 18
accuracy 0.9091 770
macro avg 0.8983 0.8782 0.8816 770
weighted avg 0.9143 0.9091 0.9076 770
Bert_CNN_LSTM_Model
from kashgari.tasks.classification import CNN_LSTM_Model
model_4 = CNN_LSTM_Model(bert_embed)
model_4.fit(train_x, train_y, valid_x, valid_y,epochs=2, batch_size=64)
result_4 = model_4.evaluate(test_x, test_y)
model_4.save('model/BERT_CNN_LSTM_Model.h5')
Model: "model_7"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
Input-Token (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Input-Segment (InputLayer) [(None, 128)] 0
__________________________________________________________________________________________________
Embedding-Token (TokenEmbedding [(None, 128, 1024), 21635072 Input-Token[0][0]
__________________________________________________________________________________________________
Embedding-Segment (Embedding) (None, 128, 1024) 2048 Input-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Token-Segment (Add) (None, 128, 1024) 0 Embedding-Token[0][0]
Embedding-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Position (PositionEmb (None, 128, 1024) 131072 Embedding-Token-Segment[0][0]
__________________________________________________________________________________________________
Embedding-Dropout (Dropout) (None, 128, 1024) 0 Embedding-Position[0][0]
__________________________________________________________________________________________________
Embedding-Norm (LayerNormalizat (None, 128, 1024) 2048 Embedding-Dropout[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Embedding-Norm[0][0]
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 0 Embedding-Norm[0][0]
Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-1-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-1-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-1-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-1-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-1-MultiHeadSelfAttention-
Encoder-1-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-1-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-1-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-1-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-1-FeedForward-Norm[0][0]
Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-2-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-2-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-2-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-2-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-2-MultiHeadSelfAttention-
Encoder-2-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-2-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-2-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-2-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-2-FeedForward-Norm[0][0]
Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-3-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-3-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-3-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-3-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-3-MultiHeadSelfAttention-
Encoder-3-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-3-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-3-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-3-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-3-FeedForward-Norm[0][0]
Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-4-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-4-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-4-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-4-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-4-MultiHeadSelfAttention-
Encoder-4-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-4-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-4-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-4-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-4-FeedForward-Norm[0][0]
Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-5-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-5-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-5-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-5-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-5-MultiHeadSelfAttention-
Encoder-5-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-5-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-5-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-5-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-5-FeedForward-Norm[0][0]
Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-6-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-6-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-6-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-6-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-6-MultiHeadSelfAttention-
Encoder-6-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-6-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-6-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-6-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-6-FeedForward-Norm[0][0]
Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-7-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-7-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-7-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-7-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-7-MultiHeadSelfAttention-
Encoder-7-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-7-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-7-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-7-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-7-FeedForward-Norm[0][0]
Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-8-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-8-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-8-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-8-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-8-MultiHeadSelfAttention-
Encoder-8-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-8-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-8-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 4198400 Encoder-8-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention[
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 0 Encoder-8-FeedForward-Norm[0][0]
Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-MultiHeadSelfAttentio (None, 128, 1024) 2048 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward (FeedForw (None, 128, 1024) 8393728 Encoder-9-MultiHeadSelfAttention-
__________________________________________________________________________________________________
Encoder-9-FeedForward-Dropout ( (None, 128, 1024) 0 Encoder-9-FeedForward[0][0]
__________________________________________________________________________________________________
Encoder-9-FeedForward-Add (Add) (None, 128, 1024) 0 Encoder-9-MultiHeadSelfAttention-
Encoder-9-FeedForward-Dropout[0][
__________________________________________________________________________________________________
Encoder-9-FeedForward-Norm (Lay (None, 128, 1024) 2048 Encoder-9-FeedForward-Add[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 4198400 Encoder-9-FeedForward-Norm[0][0]
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 0 Encoder-9-FeedForward-Norm[0][0]
Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-MultiHeadSelfAttenti (None, 128, 1024) 2048 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward (FeedFor (None, 128, 1024) 8393728 Encoder-10-MultiHeadSelfAttention
__________________________________________________________________________________________________
Encoder-10-FeedForward-Dropout (None, 128, 1024) 0 Encoder-10-FeedForward[0][0]
扫一扫
1000
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
