PyTorch crop images differentiablly

最新推荐文章于 2024-12-05 17:13:29 发布

trium_KW

最新推荐文章于 2024-12-05 17:13:29 发布

阅读量488

点赞数

分类专栏：算法文章标签： pytorch 仿射变换

本文链接：https://blog.youkuaiyun.com/trium_KW/article/details/106294346

版权

算法专栏收录该内容

5 篇文章

订阅专栏

本文介绍了如何在PyTorch中进行全微分图像裁剪。首先，解释了PyTorch图像坐标系，这是一个左手法则笛卡尔坐标系，坐标归一化到[-1,1]。接着，讨论了仿射变换理论，用于从裁剪后的图像坐标系统映射到原始图像系统。最后，提供了将仿射变换参数化为矩阵Θ的代码实现，以及从裁剪图像坐标找到原始图像坐标的函数。" 112246069,10295287,Contextual Loss在图像风格迁移的应用与理解,"['图像处理', '深度学习', 'CNN', '风格迁移', '损失函数']

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Intro

PyTorch provides a variety of means to crop images. For example, torchvision.transforms provides several functions to crop PIL images; PyTorch Forum provides an answer of how to crop image in a differentiable way (differentiable with respect to the image). However, sometimes we need a fully differentiable approach for the cropping action itself. How shall we implement that?

Theory: Affine transformation

Before reaching the answer, we need first to learn about the image coordinate system in PyTorch. It is a left-handed Cartesian system origined at the middle of an image. The coordinate has been normalized to range $[- 1, 1]$ , where $(- 1, - 1)$ indicates the top-left corner, and $(1, 1)$ indicates the bottom-right corner, as pointed out by the doc.

Let $(x, y)$ be the top-left corner of the cropped image with respect to the coordinate of the original image; likewise, we denote $(x^{'}, y^{'})$ as the bottom-right corner of the cropped image. It’s clear that $(x, y)$ corresponds to $(- 1, - 1)$ with respect to the cropped image coordinate system, and $(x^{'}, y^{'})$ corresponds to $(1, 1)$ . We’d like a function $f$ that maps from the cropped image system to the original image system for every point in the cropped image. Since only scaling and translation are involved, the function $f$ can be parameterized by an affine transformation matrix $\Theta$ such that

$\Theta = \begin{pmatrix} \theta_{11} & 0 & \theta_{13}\\ 0 & \theta_{22} & \theta_{23}\\ 0 & 0 & 1\\ \end{pmatrix}$

where $\theta_{12}=\theta_{21}=0$ since skewing is not involved. Denote $\mathbf{u}_H$ as the homogeneous coordinate of $\mathbf{u}=\begin{pmatrix}u & v\\ \end{pmatrix}^T$ such that $\mathbf{u}_H=\begin{pmatrix}\mathbf{u}^T&1\end{pmatrix}^T$ , $\Theta$ maps $\mathbf{u}_H$ with respect to the cropped image system to $\mathbf{x}_H$ with respect to the original image system, i.e. $\mathbf{x}_H = \Theta \mathbf{u}_H$ . Thus,

$\begin{pmatrix} x & x'\\ y & y'\\ 1 & 1 \end{pmatrix} = \begin{pmatrix} \theta_{11} & 0 & \theta_{13}\\ 0 & \theta_{22} & \theta_{23}\\ 0 & 0 & 1\\ \end{pmatrix} \begin{pmatrix} -1 & 1\\ -1 & 1\\ 1 & 1\\ \end{pmatrix}$

Solving the equations,

$\Theta = \begin{pmatrix} \frac{x'-x}{2} & 0 & \frac{x'+x}{2}\\ 0 & \frac{y'-y}{2} & \frac{y'+y}{2}\\ 0 & 0 & 1\\ \end{pmatrix}$

where $x'\ge x, y' \ge y$ .

Coding time

We’ll need two functions:

torch.nn.functional.affine_grid to convert the $\Theta$ parameterization to $f$
torch.nn.functional.grid_sample to find the corresponding original image coordinate from each cropped image coordinate

import torch
import torch.nn.functional as F

B, C, H, W = 16, 3, 224, 224  # batch size, input channels
                              # original image height and width
# Let `I` be our original image
I = torch.rand(B, C, H, W)
# Set the (x,y) and (x',y') to define the rectangular region to crop
x, y = -0.5, -0.3  # some examplary random coordinates;
x_, y_ = 0.7, 0.8  # in practice, (x,y,x_,y_) might be predicted
                   # as a tensor in the computation graph
# Set the affine parameters
theta = torch.tensor([
    [(x_-x)/2,       0, (x_+x)/2],
    [       0,(y_-y)/2, (y_+y)/2],
]).unsqueeze_(0).expand(B, -1, -1)
# compute the flow field;
# where size is the output size (scaling involved)
# `align_corners` option must be the same throughout the code
f = F.affine_grid(theta, size=(B, C, H//2, W//2), align_corners=False)
I_cropped = F.grid_sample(I, f, align_corners=False)