1. Basic Functionality

  • With known camera intrinsic and extrinsic parameters, obtain both the infrared depth map and the real depth map of an object, convert them into point clouds, and visualize the results.

2. Implementation Process

2.1 Initialize the Environment

nd_main_intrinsic_matrix = np.array([[600, 0, 240],
                                     [0, 600, 240],
                                     [0, 0, 1]])
nd_ir_intrinsic_matrix = np.array([[480, 0, 240],
                                   [0, 480, 240],
                                   [0, 0, 1]])

env = RFUniverseBaseEnv(scene_file="ActiveDepth.json", ext_attr=[ActiveLightSensorAttr])
active_light_sensor_1 = env.GetAttr(789789)
  • Load the environment from a json file.

  • Initialize the camera intrinsic matrices.

  • Here, $f_x, f_y$ are equal, representing an ideal pinhole camera scenario.

2.2 Obtain Images

2.2.1 Obtain RGB Image

image_byte = active_light_sensor_1.data["rgb"]
image_rgb = np.frombuffer(image_byte, dtype=np.uint8)
image_rgb = cv2.imdecode(image_rgb, cv2.IMREAD_COLOR)
image_rgb = cv2.cvtColor(image_rgb, cv2.COLOR_BGR2RGB)

2.2.2 Obtain Object Semantic Segmentation Map

image_id = active_light_sensor_1.data["id_map"]
image_id = np.frombuffer(image_id, dtype=np.uint8)
image_id = cv2.imdecode(image_id, cv2.IMREAD_COLOR)
image_id = cv2.cvtColor(image_id, cv2.COLOR_BGR2RGB)

2.2.3 Obtain Depth Map

image_depth_exr = active_light_sensor_1.data["depth_exr"]

image_active_depth = active_light_sensor_1.data["active_depth"]
  • GetDepthEXR returns the depth map in EXR format, considered to be without error.

  • GetActiveDepth simulates a real-world infrared depth camera, returning a depth map with errors.

2.3 Point Cloud Rendering

2.3.1 Obtain the Camera’s Extrinsic Matrix

local_to_world_matrix = active_light_sensor_1.data["local_to_world_matrix"]
  • local_to_world_matrix: The coordinate transformation matrix from camera local to world coordinate system, i.e., the camera’s extrinsic matrix.

2.3.2 Render Point Cloud Using the Error-Free Depth Map

real_point_cloud1 = dp.image_bytes_to_point_cloud_intrinsic_matrix(
    image_byte, image_depth_exr, nd_main_intrinsic_matrix, local_to_world_matrix
  • image_bytes_to_point_cloud_intrinsic_matrix: Generates a global coordinate system point cloud using the original byte-format RGB and depth images, along with the camera’s intrinsic and extrinsic matrices.

2.3.3 Render Point Cloud Using the Depth Map with Errors

active_point_cloud1 = dp.image_array_to_point_cloud_intrinsic_matrix(
    image_rgb, image_active_depth, nd_main_intrinsic_matrix, local_to_world_matrix
  • image_array_to_point_cloud_intrinsic_matrix: Generates a global coordinate system point cloud using numpy.ndarray-format RGB and depth images, along with the camera’s intrinsic and extrinsic matrices.

2.3.4 Segment Objects in the Point Cloud

color = utility.EncodeIDAsColor(568451)[0:3]
mask_real_point_cloud1 = dp.mask_point_cloud_with_id_color(
    real_point_cloud1, image_id, color
mask_active_point_cloud1 = dp.mask_point_cloud_with_id_color(
    active_point_cloud1, image_id, color
  • mask_point_cloud_with_id_color: Uses the previously generated object semantic segmentation map to segment target objects from the point cloud.

2.3.5 Filter the Point Cloud with Errors for Improvement

filtered_point_cloud1 = dp.filter_active_depth_point_cloud_with_exact_depth_point_cloud(
    mask_active_point_cloud1, mask_real_point_cloud1
  • filter_active_depth_point_cloud_with_exact_depth_point_cloud: Uses the real depth map to filter and improve the point cloud generated from the depth map with errors.

2.3.6 Repeat the Process for Another Viewpoint Camera

active_light_sensor_2 = env.GetAttr(123123)

# Repeat the image capture and point cloud rendering steps for the second camera
# Finalize and close the environment
  • Repeat the above process for another viewpoint camera.

2.4 Visualization

# Transform from unity space to open3d space and display
# Apply transformations and visualize the different point clouds

coordinate = o3d.geometry.TriangleMesh.create_coordinate_frame()

o3d.visualization.draw_geometries([real_point_cloud1, real_point_cloud2, coordinate])
o3d.visualization.draw_geometries([active_point_cloud1, active_point_cloud2, coordinate])
o3d.visualization.draw_geometries([mask_real_point_cloud1, mask_real_point_cloud2, coordinate])
o3d.visualization.draw_geometries([mask_active_point_cloud1, mask_active_point_cloud2, coordinate])
o3d.visualization.draw_geometries([filtered_point_cloud1, filtered_point_cloud2, coordinate])

This tutorial segment demonstrates the process to obtain and visualize depth maps and point clouds, highlighting the capabilities of the pyrfuniverse simulation platform in handling infrared depth and realistic visualization.