Point Cloud from Images Examples
This example will guide you through test_point_cloud.py that captures images from cameras in RFUniverse and converts them into point clouds using
Open3D libraries. The code further transforms the point clouds into an appropriate coordinate frame and visualizes them.
Understanding the Code
The consists of several sections. Below we will discuss each part of the code line by line.
The first line of code enable OpenEXR files in OpenCV. The last few lines import the Open3D library and raise an ImportError if it is not installed.
import os os.environ["OPENCV_IO_ENABLE_OPENEXR"] = "1" from pyrfuniverse.envs.base_env import RFUniverseBaseEnv import pyrfuniverse.utils.depth_processor as dp import numpy as np try: import open3d as o3d except ImportError: print('This feature requires open3d, please install with `pip install open3d`') raise
Creating a Simulation Environment Instance and Capturing Images:
The next code creates an instance of the RFUniverse simulation environment by loading a predefined JSON file describing the scene and captures images from cameras in the environment.
env = RFUniverseBaseEnv(scene_file='PointCloud.json') camera1 = env.GetAttr(698548) camera1.GetDepthEXR(width=1920, height=1080) camera1.GetRGB(width=1920, height=1080) camera1.GetID(width=1920, height=1080)
Transform the RGBD image into point cloud:
The next code set gets rdb image and depth image from camera, as well as corresponding transformation matrix to calculate the point cloud in the camera’s coordinate.
env.step() image_rgb = camera1.data['rgb'] image_depth_exr = camera1.data['depth_exr'] fov = camera1.data['fov'] local_to_world_matrix = camera1.data['local_to_world_matrix'] local_to_world_matrix = np.reshape(local_to_world_matrix, [4, 4]).T point1 = dp.image_bytes_to_point_cloud(image_rgb, image_depth_exr, fov, local_to_world_matrix)
Calculate the point cloud from the second camera.
The next code set is the same process as the above. For the second camera in this scene, calculate the point cloud in its coordinate.
camera2 = env.GetAttr(698550) camera2.GetDepthEXR(width=1920, height=1080) camera2.GetRGB(width=1920, height=1080) camera2.GetID(width=1920, height=1080) env.step() image_rgb = camera2.data['rgb'] image_depth_exr = camera2.data['depth_exr'] fov = camera2.data['fov'] local_to_world_matrix = camera2.data['local_to_world_matrix'] local_to_world_matrix = np.reshape(local_to_world_matrix, [4, 4]).T point2 = dp.image_bytes_to_point_cloud(image_rgb, image_depth_exr, fov, local_to_world_matrix) env.close()
Transforming the points from Unity Coordinate to Open3d Coordinate:
The next few lines of code transform the points. The “transform” method applies the necessary transformation to bring the point clouds into a suitable coordinate frame.
point1.transform([[-1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]) point2.transform([[-1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
Visualizing the Point Clouds:
The final lines of code visualize the point clouds using Open3D’s visualization module.
coorninate = o3d.geometry.TriangleMesh.create_coordinate_frame() o3d.visualization.draw_geometries([point1, point2, coorninate])