（6）点云数据处理学习——RGBD图

chencaw

已于 2022-12-02 11:23:43 修改

阅读量3.9k

点赞数 8

CC 4.0 BY-SA版权

分类专栏：深度学习文章标签：学习

于 2022-12-02 10:19:13 首次发布

本文链接：https://blog.youkuaiyun.com/chencaw/article/details/128142976

深度学习专栏收录该内容

53 篇文章

订阅专栏

本文介绍如何使用Open3D库从不同RGBD数据集（包括Redwood、SUN、NYU和TUM）读取和处理RGBD图像。通过具体实例展示了如何创建RGBD图像对象、显示图像以及将其转换为点云。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、主要参考

（1）官网的地址

RGBD images — Open3D 0.16.0 documentation

2、实现和测试

2.1 功能说明

Open3D有一个用于图像的数据结构。它支持各种函数，如read_image、write_image、filter_image和draw_geometries。一个Open3D图像可以直接转换为numpy数组。

一个Open3D RGBDImage由两个图像rgbimage .depth和rgbimage .color组成。我们要求两个图像注册到相同的相机帧和具有相同的分辨率。下面的教程展示了如何从许多著名的RGBD数据集中读取和使用RGBD图像。

2.2Redwood 数据集（Redwood dataset）

（1）数据集说明

在本节中，我们将展示如何从Redwood数据集[Choi2015]_读取和可视化RGBDImage。

Redwood格式在16位单通道图像中存储深度。整数值表示以毫米为单位的深度测量。它是Open3D解析深度图像的默认格式。

（2）使用的函数

create_from_color_and_depth( color_raw, depth_raw)

功能说明：默认的转换函数create_rgbd_image_from_color_and_depth从一对彩色和深度图像创建一个RGBDImage。彩色图像被转换为灰度图像，存储在浮动范围[0,1]中。深度图像存储在float中，以米为单位表示深度值。

（3）测试代码

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt


print("Read Redwood dataset")
redwood_rgbd = o3d.data.SampleRedwoodRGBDImages()
color_raw = o3d.io.read_image(redwood_rgbd.color_paths[0])
depth_raw = o3d.io.read_image(redwood_rgbd.depth_paths[0])
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
    color_raw, depth_raw)
print(rgbd_image)

注意open3d自动下载解压后在下面位置

几张椅子的图挺漂亮的

2.2.1显示一下RGB图和深度图

（1）可以通过np.asarray直接对深度图和彩色图进行处理

color_img = np.asarray(rgbd_image.color)
depth_img = np.asarray(rgbd_image.depth)
print(np.size(color_img))
print(color_img[0][0])
print(depth_img[0][0])

（2）测试代码

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt


print("Read Redwood dataset")
redwood_rgbd = o3d.data.SampleRedwoodRGBDImages()
color_raw = o3d.io.read_image(redwood_rgbd.color_paths[0])
depth_raw = o3d.io.read_image(redwood_rgbd.depth_paths[0])
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
    color_raw, depth_raw)
print(rgbd_image)

#640*480的图
color_img = np.asarray(rgbd_image.color)
depth_img = np.asarray(rgbd_image.depth)
print(np.size(color_img))
print(color_img[0][0])
print(depth_img[0][0])


plt.subplot(1, 2, 1)
plt.title('Redwood grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('Redwood depth image')
plt.imshow(rgbd_image.depth)
plt.show()

（3）测试的结果

2.2.2RGBD图像可以转换为点云图

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt


print("Read Redwood dataset")
redwood_rgbd = o3d.data.SampleRedwoodRGBDImages()
color_raw = o3d.io.read_image(redwood_rgbd.color_paths[0])
depth_raw = o3d.io.read_image(redwood_rgbd.depth_paths[0])
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
    color_raw, depth_raw)
print(rgbd_image)

#640*480的图
# color_img = np.asarray(rgbd_image.color)
# depth_img = np.asarray(rgbd_image.depth)
# print(np.size(color_img))
# print(color_img[0][0])
# print(depth_img[0][0])


# plt.subplot(1, 2, 1)
# plt.title('Redwood grayscale image')
# plt.imshow(rgbd_image.color)
# plt.subplot(1, 2, 2)
# plt.title('Redwood depth image')
# plt.imshow(rgbd_image.depth)
# plt.show()

pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
    rgbd_image,
    o3d.camera.PinholeCameraIntrinsic(
        o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.5)

o3d.visualization.draw_geometries([pcd])

这里使用了PinholeCameraIntrinsicParameters。PrimeSenseDefault作为默认相机参数。它的图像分辨率为640x480，焦距(fx, fy) =(525.0, 525.0)，光心(cx, cy) =(319.5, 239.5)。单位矩阵被用作默认的外部参数。pcd.transform在点云上应用上下翻转转换，以达到更好的可视化目的。

2.3 SUN数据集(SUN dataset)

（1）说明

在本节中，我们将展示如何读取和可视化SUN数据集[Song2015]_的RGBDImage。

本教程与上面处理Redwood数据集的教程几乎相同。唯一的区别是我们使用了转换函数create_rgbd_image_from_sun_format来解析SUN数据集中的深度图像。

（2）测试

1）下载后的数据集如下图所示

2）测试代码

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt



print("Read SUN dataset")
sun_rgbd = o3d.data.SampleSUNRGBDImage()
color_raw = o3d.io.read_image(sun_rgbd.color_path)
depth_raw = o3d.io.read_image(sun_rgbd.depth_path)
rgbd_image = o3d.geometry.RGBDImage.create_from_sun_format(color_raw, depth_raw)
print(rgbd_image)


plt.subplot(1, 2, 1)
plt.title('SUN grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('SUN depth image')
plt.imshow(rgbd_image.depth)
plt.show()

pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
    rgbd_image,
    o3d.camera.PinholeCameraIntrinsic(
        o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.5)
o3d.visualization.draw_geometries([pcd])

转变成的点云图如下图所示

2.4 NYU数据集(NYU dataset)

本部分展示如何从NYU数据集[Silberman2012]_读取和可视化RGBDImage。

（1）说明

本部分教程与上面处理Redwood数据集的教程几乎相同，但有两点不同。首先，纽约大学的图片不是标准的jpg或png格式。因此，我们使用mpimg。imread将彩色图像作为numpy数组读取，并将其转换为Open3D图像。调用另一个辅助函数read_nyu_pgm从NYU数据集中使用的特殊大端pgm格式中读取深度图像。其次，我们使用一个不同的转换函数create_rgbd_image_from_nyu_format来解析SUN数据集中的深度图像

（2）测试代码

import open3d as o3d
import numpy as np

import matplotlib.image as mpimg
import re


# This is special function used for reading NYU pgm format
# as it is written in big endian byte order.
def read_nyu_pgm(filename, byteorder='>'):
    with open(filename, 'rb') as f:
        buffer = f.read()
    try:
        header, width, height, maxval = re.search(
            b"(^P5\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
    except AttributeError:
        raise ValueError("Not a raw PGM file: '%s'" % filename)
    img = np.frombuffer(buffer,
                        dtype=byteorder + 'u2',
                        count=int(width) * int(height),
                        offset=len(header)).reshape((int(height), int(width)))
    img_out = img.astype('u2')
    return img_out


print("Read NYU dataset")
# Open3D does not support ppm/pgm file yet. Not using o3d.io.read_image here.
# MathplotImage having some ISSUE with NYU pgm file. Not using imread for pgm.
nyu_rgbd = o3d.data.SampleNYURGBDImage()
color_raw = mpimg.imread(nyu_rgbd.color_path)
depth_raw = read_nyu_pgm(nyu_rgbd.depth_path)
color = o3d.geometry.Image(color_raw)
depth = o3d.geometry.Image(depth_raw)
rgbd_image = o3d.geometry.RGBDImage.create_from_nyu_format(color, depth)
print(rgbd_image)

下载解压后的图片如下图所示

（3）显示一下

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
import matplotlib.image as mpimg
import re


# This is special function used for reading NYU pgm format
# as it is written in big endian byte order.
def read_nyu_pgm(filename, byteorder='>'):
    with open(filename, 'rb') as f:
        buffer = f.read()
    try:
        header, width, height, maxval = re.search(
            b"(^P5\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
    except AttributeError:
        raise ValueError("Not a raw PGM file: '%s'" % filename)
    img = np.frombuffer(buffer,
                        dtype=byteorder + 'u2',
                        count=int(width) * int(height),
                        offset=len(header)).reshape((int(height), int(width)))
    img_out = img.astype('u2')
    return img_out


print("Read NYU dataset")
# Open3D does not support ppm/pgm file yet. Not using o3d.io.read_image here.
# MathplotImage having some ISSUE with NYU pgm file. Not using imread for pgm.
nyu_rgbd = o3d.data.SampleNYURGBDImage()
color_raw = mpimg.imread(nyu_rgbd.color_path)
depth_raw = read_nyu_pgm(nyu_rgbd.depth_path)
color = o3d.geometry.Image(color_raw)
depth = o3d.geometry.Image(depth_raw)
rgbd_image = o3d.geometry.RGBDImage.create_from_nyu_format(color, depth)
print(rgbd_image)

plt.subplot(1, 2, 1)
plt.title('NYU grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('NYU depth image')
plt.imshow(rgbd_image.depth)
plt.show()

（4）转换为点云图像显示

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt
import matplotlib.image as mpimg
import re


# This is special function used for reading NYU pgm format
# as it is written in big endian byte order.
def read_nyu_pgm(filename, byteorder='>'):
    with open(filename, 'rb') as f:
        buffer = f.read()
    try:
        header, width, height, maxval = re.search(
            b"(^P5\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
    except AttributeError:
        raise ValueError("Not a raw PGM file: '%s'" % filename)
    img = np.frombuffer(buffer,
                        dtype=byteorder + 'u2',
                        count=int(width) * int(height),
                        offset=len(header)).reshape((int(height), int(width)))
    img_out = img.astype('u2')
    return img_out


print("Read NYU dataset")
# Open3D does not support ppm/pgm file yet. Not using o3d.io.read_image here.
# MathplotImage having some ISSUE with NYU pgm file. Not using imread for pgm.
nyu_rgbd = o3d.data.SampleNYURGBDImage()
color_raw = mpimg.imread(nyu_rgbd.color_path)
depth_raw = read_nyu_pgm(nyu_rgbd.depth_path)
color = o3d.geometry.Image(color_raw)
depth = o3d.geometry.Image(depth_raw)
rgbd_image = o3d.geometry.RGBDImage.create_from_nyu_format(color, depth)
print(rgbd_image)

plt.subplot(1, 2, 1)
plt.title('NYU grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('NYU depth image')
plt.imshow(rgbd_image.depth)
plt.show()


pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
    rgbd_image,
    o3d.camera.PinholeCameraIntrinsic(
        o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.5)
o3d.visualization.draw_geometries([pcd])

2.5 TUM数据集(TUM dataset)

（1）说明

本节展示如何从TUM数据集[Strum2012]_读取和可视化RGBDImage。

本教程与上面处理Redwood数据集的教程几乎相同。唯一的区别是我们使用转换函数create_rgbd_image_from_tum_format来解析TUM数据集中的深度图像。

（2）测试

注意：官网教程有误

# tum_rgbd = o3d.data.SampleSUNRGBDImage()

tum_rgbd = o3d.data.SampleTUMRGBDImage()

或者手动下载

https://github.com/isl-org/open3d_downloads/releases/download/20220201-data/SampleTUMRGBDImage.zip

import open3d as o3d
import numpy as np
from matplotlib import pyplot as plt


print("Read TUM dataset")
# 方法一：官网下载
#tum_rgbd = o3d.data.SampleTUMRGBDImage()
# color_raw = o3d.io.read_image(tum_rgbd.color_path)
# depth_raw = o3d.io.read_image(tum_rgbd.depth_path)
## 方法二 本地下载后使用 https://github.com/isl-org/open3d_downloads/releases/download/20220201-data/SampleTUMRGBDImage.zip 
tum_rgbd_color_path = "D:/RGBD_CAMERA/python_3d_process/SampleTUMRGBDImage/TUM_color.png"
tum_rgbd_depth_path = "D:/RGBD_CAMERA/python_3d_process/SampleTUMRGBDImage/TUM_depth.png"
color_raw = o3d.io.read_image(tum_rgbd_color_path)
depth_raw = o3d.io.read_image(tum_rgbd_depth_path)
rgbd_image = o3d.geometry.RGBDImage.create_from_tum_format(color_raw, depth_raw)
print(rgbd_image)

plt.subplot(1, 2, 1)
plt.title('TUM grayscale image')
plt.imshow(rgbd_image.color)
plt.subplot(1, 2, 2)
plt.title('TUM depth image')
plt.imshow(rgbd_image.depth)
plt.show()

pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
    rgbd_image,
    o3d.camera.PinholeCameraIntrinsic(
        o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))
# # Flip it, otherwise the pointcloud will be upside down
# pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
# o3d.visualization.draw_geometries([pcd], zoom=0.35)
o3d.visualization.draw_geometries([pcd])

转换后的点云图像如下