kernel_regularizerand Activity regularizers

最新推荐文章于 2024-11-13 18:04:09 发布

原创最新推荐文章于 2024-11-13 18:04:09 发布 · 399 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#tensorflow

正则化专栏收录该内容

1 篇文章

订阅专栏

本文介绍了深度学习中正则化的概念，特别是TensorFlow库中的kernel_regularizer、bias_regularizer和activity_regularizer。通过实例展示了如何在Dense层中应用L1_L2正则化，并解释了它们分别作用于权重、偏置和输出的效果。正则化有助于防止过拟合，保持模型的泛化能力。

kernel_regularizer: Regularizer to apply a penalty on the layer’s
kernel bias_regularizer: Regularizer to apply a penalty on the
layer’s bias activity_regularizer: Regularizer to apply a penalty on the layer’s output

#coding=utf-8

import tensorflow as tf

layer = tf.keras.layers.Dense(units=5,
                              kernel_initializer='ones',
                              kernel_regularizer=tf.keras.regularizers.l1_l2(l1=0.01, l2=0.01))

tensor = tf.ones(shape=(5, 5)) * 2.0
# print("tensor:", tensor)
out = layer(tensor)
print("out:", out)
print("----start----")
print("trainable_variables:", layer.trainable_variables)
print("----end------")
# The kernel regularization term is 0.25 0.25
print(tf.math.reduce_sum(layer.losses))  # 0.5 (= 0.25 + 0.25)

out: tf.Tensor(
[[10. 10. 10. 10. 10.]
 [10. 10. 10. 10. 10.]
 [10. 10. 10. 10. 10.]
 [10. 10. 10. 10. 10.]
 [10. 10. 10. 10. 10.]], shape=(5, 5), dtype=float32)
----start----
trainable_variables: [<tf.Variable 'dense/kernel:0' shape=(5, 5) dtype=float32, numpy=
array([[1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1.]], dtype=float32)>, <tf.Variable 'dense/bias:0' shape=(5,) dtype=float32, numpy=array([0., 0., 0., 0., 0.], dtype=float32)>]
----end------
tf.Tensor(0.5, shape=(), dtype=float32)

The activity regularizer works as a function of the output of the net, and is mostly used to regularize hidden units, while weight_regularizer, as the name says, works on the weights (e.g. making them decay). Basically you can express the regularization loss as a function of the output (activity_regularizer) or of the weights (weight_regularizer).

Keras: Difference between Kernel and Activity regularizers
正则化器的使用
 Layer weight regularizers