T10猫狗识别之数据增强

原创 于 2024-12-20 14:15:04 发布 · 902 阅读 · 13 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#tensorflow #人工智能 #python

  • 🍨 本文为🔗365天深度学习训练营 中的学习记录博客
  • 🍖 原作者：K同学啊
    我的环境
    语言环境：python 3.7.12
    编译器：pycharm
    深度学习环境：tensorflow 2.7.0
    数据：本地数据集34-data

一、总结

这次实验除了训练集、验证集还设置了测试集。

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.3,
    subset="training",
    seed=12,
    image_size=(img_height, img_width),
    batch_size=batch_size)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.3,
    subset="validation",
    seed=12,
    image_size=(img_height, img_width),
    batch_size=batch_size)

#由于原始数据集不包含测试集，因此需要创建一个。
# 使用 tf.data.experimental.cardinality 确定验证集中有多少批次的数据，然后将其中的 20% 移至测试集。
val_batches = tf.data.experimental.cardinality(val_ds)
test_ds     = val_ds.take(val_batches // 5)
val_ds      = val_ds.skip(val_batches // 5)

print('Number of validation batches: %d' % tf.data.experimental.cardinality(val_ds))
print('Number of test batches: %d' % tf.data.experimental.cardinality(test_ds))

数据增强的两种方式

方法一：将其嵌入model中

model = tf.keras.Sequential([
  data_augmentation,
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
])

优点：
● 数据增强这块的工作可以得到GPU的加速（如果你使用了GPU训练的话）
注意：只有在模型训练时（Model.fit）才会进行增强，在模型评估(Model.evaluate)以及预测(Model.predict)时并不会进行增强操作。

方法二：在Dataset数据集中进行数据增强

batch_size = 32
AUTOTUNE = tf.data.AUTOTUNE

def prepare(ds):
    ds = ds.map(lambda x, y: (data_augmentation(x, training=True), y), num_parallel_calls=AUTOTUNE)
    return ds

train_ds = prepare(train_ds)

自定义数据增强

import random
# 这是大家可以自由发挥的一个地方
def aug_img(image):
    seed = (random.randint(0,9), 0)
    # 随机改变图像对比度
    stateless_random_brightness = tf.image.stateless_random_contrast(image, lower=0.1, upper=1.0, seed=seed)
    return stateless_random_brightness

image = tf.expand_dims(images[3]*255, 0)
print("Min and max pixel values:", image.numpy().min(), image.numpy().max())

plt.figure(figsize=(8, 8))
for i in range(9):
    augmented_image = aug_img(image)
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(augmented_image[0].numpy().astype("uint8"))

    plt.axis("off")

那么如何将自定义增强函数应用到我们数据上呢？请参考下文的 preprocess_image 函数，将 aug_img 函数嵌入到 preprocess_image 函数中，在数据预处理时完成数据增强就OK啦。

# #方式二：在Dataset数据集中进行数据增强
batch_size = 32
AUTOTUNE = tf.data.AUTOTUNE

def prepare(ds):
    # ds = ds.map(lambda x, y: (data_augmentation(x, training=True), y), num_parallel_calls=AUTOTUNE)
    ds = ds.map(lambda x, y: (aug_img(x), y), num_parallel_calls=AUTOTUNE)
    return ds
train_ds = prepare(train_ds)

在这里插入图片描述
模型训练与预测结果
14/14 [] - 120s 5s/step - loss: 1.8951 - accuracy: 0.4465 - val_loss: 0.6816 - val_accuracy: 0.6486
Epoch 2/20
14/14 [] - 0s 18ms/step - loss: 0.6815 - accuracy: 0.5534 - val_loss: 0.6288 - val_accuracy: 0.5878
Epoch 3/20
14/14 [] - 0s 18ms/step - loss: 0.6291 - accuracy: 0.6146 - val_loss: 0.5057 - val_accuracy: 0.8108
Epoch 4/20
14/14 [] - 0s 18ms/step - loss: 0.4853 - accuracy: 0.8249 - val_loss: 0.3718 - val_accuracy: 0.8311
Epoch 5/20
14/14 [] - 0s 18ms/step - loss: 0.3221 - accuracy: 0.8870 - val_loss: 0.3582 - val_accuracy: 0.8649
Epoch 6/20
14/14 [] - 0s 17ms/step - loss: 0.2188 - accuracy: 0.9288 - val_loss: 0.3413 - val_accuracy: 0.8649
Epoch 7/20
14/14 [] - 0s 19ms/step - loss: 0.1607 - accuracy: 0.9643 - val_loss: 0.3995 - val_accuracy: 0.8649
Epoch 8/20
14/14 [] - 0s 19ms/step - loss: 0.1356 - accuracy: 0.9624 - val_loss: 0.3174 - val_accuracy: 0.8986
Epoch 9/20
14/14 [] - 0s 19ms/step - loss: 0.0998 - accuracy: 0.9643 - val_loss: 0.2877 - val_accuracy: 0.9189
Epoch 10/20
14/14 [] - 0s 19ms/step - loss: 0.0718 - accuracy: 0.9736 - val_loss: 0.3657 - val_accuracy: 0.9054
Epoch 11/20
14/14 [] - 0s 20ms/step - loss: 0.0457 - accuracy: 0.9969 - val_loss: 0.6842 - val_accuracy: 0.8649
Epoch 12/20
14/14 [] - 0s 19ms/step - loss: 0.0944 - accuracy: 0.9642 - val_loss: 0.3745 - val_accuracy: 0.9122
Epoch 13/20
14/14 [] - 0s 27ms/step - loss: 0.0387 - accuracy: 0.9959 - val_loss: 0.4615 - val_accuracy: 0.9054
Epoch 14/20
14/14 [] - 0s 19ms/step - loss: 0.0524 - accuracy: 0.9806 - val_loss: 0.4024 - val_accuracy: 0.9257
Epoch 15/20
14/14 [] - 0s 19ms/step - loss: 0.0088 - accuracy: 0.9985 - val_loss: 0.3981 - val_accuracy: 0.9392
Epoch 16/20
14/14 [] - 0s 19ms/step - loss: 0.0103 - accuracy: 1.0000 - val_loss: 0.4682 - val_accuracy: 0.9189
Epoch 17/20
14/14 [] - 0s 18ms/step - loss: 0.0156 - accuracy: 0.9995 - val_loss: 0.4237 - val_accuracy: 0.9324
Epoch 18/20
14/14 [] - 0s 18ms/step - loss: 0.0196 - accuracy: 0.9974 - val_loss: 0.4998 - val_accuracy: 0.9122
Epoch 19/20
14/14 [] - 0s 19ms/step - loss: 0.0521 - accuracy: 0.9944 - val_loss: 0.6491 - val_accuracy: 0.8919
Epoch 20/20
14/14 [] - 0s 18ms/step - loss: 0.0282 - accuracy: 0.9951 - val_loss: 0.5771 - val_accuracy: 0.9324
1/1 [==============================] - 0s 427ms/step - loss: 0.3045 - accuracy: 0.9062
Accuracy 0.90625

Process finished with exit code 0

二、代码

import matplotlib.pyplot as plt
import numpy as np
#隐藏警告
import warnings
warnings.filterwarnings('ignore')

from tensorflow.keras import layers
import tensorflow as tf
gpus = tf.config.list_physical_devices("GPU")

if gpus:
    tf.config.experimental.set_memory_growth(gpus[0], True)  #设置GPU显存用量按需使用
    tf.config.set_visible_devices([gpus[0]],"GPU")

# 打印显卡信息，确认GPU可用
print(gpus)
data_dir   = "./34-data"
img_height = 224
img_width  = 224
batch_size = 32

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.3,
    subset="training",
    seed=12,
    image_size=(img_height, img_width),
    batch_size=batch_size)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.3,
    subset="validation",
    seed=12,
    image_size=(img_height, img_width),
    batch_size=batch_size)

#由于原始数据集不包含测试集，因此需要创建一个。
# 使用 tf.data.experimental.cardinality 确定验证集中有多少批次的数据，然后将其中的 20% 移至测试集。
val_batches = tf.data.experimental.cardinality(val_ds)
test_ds     = val_ds.take(val_batches // 5)
val_ds      = val_ds.skip(val_batches // 5)

print('Number of validation batches: %d' % tf.data.experimental.cardinality(val_ds))
print('Number of test batches: %d' % tf.data.experimental.cardinality(test_ds))

class_names = train_ds.class_names
print(class_names)# 打印类别

AUTOTUNE = tf.data.AUTOTUNE

def preprocess_image(image,label):
    return (image/255.0,label)

# 归一化处理
train_ds = train_ds.map(preprocess_image, num_parallel_calls=AUTOTUNE)
val_ds   = val_ds.map(preprocess_image, num_parallel_calls=AUTOTUNE)
test_ds  = test_ds.map(preprocess_image, num_parallel_calls=AUTOTUNE)

train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)
val_ds   = val_ds.cache().prefetch(buffer_size=AUTOTUNE)

plt.figure(figsize=(15, 10))  # 图形的宽为15高为10

for images, labels in train_ds.take(1):
    for i in range(8):
        ax = plt.subplot(5, 8, i + 1)
        plt.imshow(images[i])
        plt.title(class_names[labels[i]])

        plt.axis("off")

plt.show()

##数据增强
data_augmentation = tf.keras.Sequential([
  tf.keras.layers.experimental.preprocessing.RandomFlip("horizontal_and_vertical"),
  tf.keras.layers.experimental.preprocessing.RandomRotation(0.2),
])

#第一个层表示进行随机的水平和垂直翻转，而第二个层表示按照 0.2 的弧度值进行随机旋转。
# Add the image to a batch.
image = tf.expand_dims(images[i], 0)
plt.figure(figsize=(8, 8))
for i in range(9):
    augmented_image = data_augmentation(image)
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(augmented_image[0])
    plt.axis("off")
plt.show()

#增强方式一：将其嵌入model中
model = tf.keras.Sequential([
  data_augmentation,
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
])
# 方式一这样做的好处是：
# ● 数据增强这块的工作可以得到GPU的加速（如果你使用了GPU训练的话）
# 注意：只有在模型训练时（Model.fit）才会进行增强，在模型评估(Model.evaluate)以及预测(Model.predict)时并不会进行增强操作。

# #方式二：在Dataset数据集中进行数据增强
# batch_size = 32
# AUTOTUNE = tf.data.AUTOTUNE
#
# def prepare(ds):
#     ds = ds.map(lambda x, y: (data_augmentation(x, training=True), y), num_parallel_calls=AUTOTUNE)
#     return ds
# train_ds = prepare(train_ds)

#训练模型
model = tf.keras.Sequential([
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(len(class_names))
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

epochs=20
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)

loss, acc = model.evaluate(test_ds)
print("Accuracy", acc)

三、结果

数据集加载
在这里插入图片描述
使用 tf.data.experimental.cardinality 确定验证集中有多少批次的数据，然后将其中的 20% 移至测试集。以及类别。
在这里插入图片描述
数据集随机输出
在这里插入图片描述
数据增强结果
在这里插入图片描述
Epoch 1/20
14/14 [] - 85s 4s/step - loss: 1.8392 - accuracy: 0.5236 - val_loss: 0.6538 - val_accuracy: 0.7905
Epoch 2/20
14/14 [] - 0s 18ms/step - loss: 0.6004 - accuracy: 0.7946 - val_loss: 0.4441 - val_accuracy: 0.8378
Epoch 3/20
14/14 [] - 0s 17ms/step - loss: 0.2920 - accuracy: 0.9256 - val_loss: 0.3296 - val_accuracy: 0.8919
Epoch 4/20
14/14 [] - 0s 18ms/step - loss: 0.1586 - accuracy: 0.9590 - val_loss: 0.3102 - val_accuracy: 0.8986
Epoch 5/20
14/14 [] - 0s 18ms/step - loss: 0.1253 - accuracy: 0.9707 - val_loss: 0.2443 - val_accuracy: 0.9324
Epoch 6/20
14/14 [] - 0s 18ms/step - loss: 0.0558 - accuracy: 0.9921 - val_loss: 0.2873 - val_accuracy: 0.9257
Epoch 7/20
14/14 [] - 0s 18ms/step - loss: 0.0249 - accuracy: 1.0000 - val_loss: 0.4234 - val_accuracy: 0.8851
Epoch 8/20
14/14 [] - 0s 17ms/step - loss: 0.0195 - accuracy: 0.9931 - val_loss: 0.3610 - val_accuracy: 0.9054
Epoch 9/20
14/14 [] - 0s 17ms/step - loss: 0.0111 - accuracy: 0.9962 - val_loss: 0.3370 - val_accuracy: 0.8986
Epoch 10/20
14/14 [] - 0s 18ms/step - loss: 0.0171 - accuracy: 1.0000 - val_loss: 0.3265 - val_accuracy: 0.9324
Epoch 11/20
14/14 [] - 0s 18ms/step - loss: 0.0301 - accuracy: 0.9965 - val_loss: 0.3381 - val_accuracy: 0.8986
Epoch 12/20
14/14 [] - 0s 18ms/step - loss: 0.0533 - accuracy: 0.9892 - val_loss: 0.3881 - val_accuracy: 0.9257
Epoch 13/20
14/14 [] - 0s 17ms/step - loss: 0.0403 - accuracy: 0.9825 - val_loss: 0.7449 - val_accuracy: 0.8514
Epoch 14/20
14/14 [] - 0s 17ms/step - loss: 0.1525 - accuracy: 0.9550 - val_loss: 0.6648 - val_accuracy: 0.8041
Epoch 15/20
14/14 [] - 0s 17ms/step - loss: 0.0935 - accuracy: 0.9710 - val_loss: 0.3407 - val_accuracy: 0.9324
Epoch 16/20
14/14 [] - 0s 18ms/step - loss: 0.0117 - accuracy: 0.9985 - val_loss: 0.6555 - val_accuracy: 0.9122
Epoch 17/20
14/14 [] - 0s 18ms/step - loss: 0.0241 - accuracy: 0.9858 - val_loss: 0.4262 - val_accuracy: 0.9459
Epoch 18/20
14/14 [] - 0s 18ms/step - loss: 0.0087 - accuracy: 0.9974 - val_loss: 0.3678 - val_accuracy: 0.9257
Epoch 19/20
14/14 [] - 0s 17ms/step - loss: 0.0188 - accuracy: 0.9915 - val_loss: 0.4021 - val_accuracy: 0.9257
Epoch 20/20
14/14 [] - 0s 17ms/step - loss: 0.0014 - accuracy: 1.0000 - val_loss: 0.4594 - val_accuracy: 0.9257
1/1 [==============================] - 1s 584ms/step - loss: 0.0517 - accuracy: 0.9688
Accuracy 0.96875

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值