scikit-surgerysurfacematch

Algorithms

ICP

PCL ICP implementation of RigidRegistration interface.

class sksurgerysurfacematch.algorithms.pcl_icp_registration.RigidRegistration(max_iterations: int = 100, max_correspondence_threshold: float = 1, transformation_epsilon: float = 0.0001, fitness_epsilon: float = 0.0001, use_lm_icp: bool = True)

Bases: sksurgerysurfacematch.interfaces.rigid_registration.RigidRegistration

Class that uses PCL implementation of ICP to register fixed/moving clouds.

register(moving_cloud: numpy.ndarray, fixed_cloud: numpy.ndarray)

Uses PCL library, wrapped in scikit-surgerypclcpp.

Parameters:

• moving_cloud – [Nx3] source/moving point cloud.
• fixed_cloud – [Mx3] target/fixed point cloud.

Returns:

[4x4] transformation matrix, moving-to-fixed space.

GoICP

Go ICP implementation of RigidRegistration interface.

class sksurgerysurfacematch.algorithms.goicp_registration.RigidRegistration(dt_size: int = 200, dt_factor: float = 2.0, normalise: bool = True, num_moving_points: int = 1000, rotation_limits=[-45, 45], trans_limits=[-0.5, 0.5])

Bases: sksurgerysurfacematch.interfaces.rigid_registration.RigidRegistration

Class that uses GoICP implementation to register fixed/moving clouds. At the moment, we are just relying on all default parameters. :param dt_size: Nodes per dimension of distance transform :param dt_factor: GoICP distance transform factor TODO: rest of params

register(moving_cloud: numpy.ndarray, fixed_cloud: numpy.ndarray) → numpy.ndarray

Uses GoICP library, wrapped in scikit-surgerygoicp.

Parameters:

• fixed_cloud – [Nx3] fixed point cloud.
• moving_cloud – [Mx3] moving point cloud.
• normalise – If true, data will be centred around 0 and normalised.
• num_moving_points – How many points to sample from moving cloud if 0, use all points

Returns:

[4x4] transformation matrix, moving-to-fixed space.

sksurgerysurfacematch.algorithms.goicp_registration.create_scaling_matrix(scale: float) → numpy.ndarray

Create a scaling matrix, with the same value in each axis.

sksurgerysurfacematch.algorithms.goicp_registration.create_translation_matrix(translate: numpy.ndarray) → numpy.ndarray

Create translation matrix from 3x1 translation vector.

sksurgerysurfacematch.algorithms.goicp_registration.demean_and_normalise(points_a: numpy.ndarray, points_b: numpy.ndarray)

Independently centre each point cloud around 0,0,0, then normalise both to [-1,1].

Parameters:

• points_a (np.ndarray) – 1st point cloud
• points_b (np.ndarray) – 2nd point cloud

Returns:

normalised points clouds, scale factor & translations

sksurgerysurfacematch.algorithms.goicp_registration.numpy_to_POINT3D_array(numpy_pointcloud)

Covert numpy array to POINT3D array suitable for GoICP algorithm.

sksurgerysurfacematch.algorithms.goicp_registration.set_rotnode(limits_degrees) → sksurgerygoicppython.ROTNODE

Setup a ROTNODE with upper/lower rotation limits

sksurgerysurfacematch.algorithms.goicp_registration.set_transnode(trans_limits) → sksurgerygoicppython.TRANSNODE

Setup a TRANSNODE with upper/lower limits

Stereo Recon Base Class

Base class for surface reconstruction on already rectified images.

class sksurgerysurfacematch.algorithms.reconstructor_with_rectified_images.StereoReconstructorWithRectifiedImages(lower_disparity_multiplier=2.0, upper_disparity_multiplier=2.0, alpha: float = 0)

Bases: sksurgerysurfacematch.interfaces.stereo_reconstructor.StereoReconstructor

Base class for those stereo reconstruction methods that work specifically from rectified images. This class handles rectification and the necessary coordinate transformations. Note: The client calls the reconstruct() method which requires undistorted images, which are NOT already rectified. It’s THIS class that does the rectification for you, and calls through to the _compute_disparity() method that derived classes must implement.

extract(left_mask: numpy.ndarray)

Extracts the actual point cloud. This is a separate method, so that you can reconstruct once using reconstruct(), and then call this extract method with multiple masks, without incurring the cost of multiple calls to the reconstruction algorithm, which may be expensive. :param left_mask: mask image, single channel, same size as left_image :return: [Nx6] point cloud where the 6 columns are x, y, z in left camera space, followed by r, g, b colours.

reconstruct(left_image: numpy.ndarray, left_camera_matrix: numpy.ndarray, right_image: numpy.ndarray, right_camera_matrix: numpy.ndarray, left_to_right_rmat: numpy.ndarray, left_to_right_tvec: numpy.ndarray, left_mask: numpy.ndarray = None)

Implementation of stereo surface reconstruction that takes undistorted images, rectifies them, asks derived classes to compute a disparity map on the rectified images, and then sorts out extracting points and their colours.

Camera parameters are those obtained from OpenCV.

Parameters:

• left_image – undistorted left image, BGR
• left_camera_matrix – [3x3] camera matrix
• right_image – undistorted right image, BGR
• right_camera_matrix – [3x3] camera matrix
• left_to_right_rmat – [3x3] rotation matrix
• left_to_right_tvec – [3x1] translation vector
• left_mask – mask image, single channel, same size as left_image

Returns:

[Nx6] point cloud where the 6 columns

are x, y, z in left camera space, followed by r, g, b colours.

SGBM Stereo Recon

Surface reconstruction using OpenCV’s SGBM reconstruction

class sksurgerysurfacematch.algorithms.sgbm_reconstructor.SGBMReconstructor(min_disparity=16, num_disparities=112, block_size=3, p_1=360, p_2=1440, disp_12_max_diff=0, uniqueness_ratio=0, speckle_window_size=0, speckle_range=0)

Bases: sksurgerysurfacematch.algorithms.reconstructor_with_rectified_images.StereoReconstructorWithRectifiedImages

Constructor. See OpenCV StereoSGBM for parameter comments.

Stoyanov Stereo Recon

Surface reconstruction using Stoyanov MICCAI 2010 paper.

class sksurgerysurfacematch.algorithms.stoyanov_reconstructor.StoyanovReconstructor(use_hartley=False)

Bases: sksurgerysurfacematch.interfaces.stereo_reconstructor.StereoReconstructor

Constructor.

reconstruct(left_image: numpy.ndarray, left_camera_matrix: numpy.ndarray, right_image: numpy.ndarray, right_camera_matrix: numpy.ndarray, left_to_right_rmat: numpy.ndarray, left_to_right_tvec: numpy.ndarray, left_mask: numpy.ndarray = None)

Implementation of dense stereo surface reconstruction using Dan Stoyanov’s MICCAI 2010 method.

Camera parameters are those obtained from OpenCV.

Parameters:

• left_image – undistorted left image, BGR
• left_camera_matrix – [3x3] camera matrix
• right_image – undistorted right image, BGR
• right_camera_matrix – [3x3] camera matrix
• left_to_right_rmat – [3x3] rotation matrix
• left_to_right_tvec – [3x1] translation vector
• left_mask – mask image, single channel, same size as left_image

Returns:

[Nx6] point cloud where the 6 columns

are x, y, z in left camera space, and r, g, b, colors.

Interfaces

Rigid Registration

Base class (pure virtual interface) for rigid registration.

class sksurgerysurfacematch.interfaces.rigid_registration.RigidRegistration

Bases: object

Base class for classes that can rigidly register (align), two point clouds.

register(source_cloud: numpy.ndarray, target_cloud: numpy.ndarray)

A derived class must implement this.

Parameters:

• source_cloud – [Nx3] fixed point cloud.
• target_cloud – [Mx3] moving point cloud.

Returns:

residual, [4x4] transformation matrix, moving-to-fixed space.

Stereo Reconstruction

Base class (pure virtual interface) for classes that do stereo recon.

class sksurgerysurfacematch.interfaces.stereo_reconstructor.StereoReconstructor

Bases: object

Base class for stereo reconstruction algorithms. Clients call the reconstruct() method, passing in undistorted images. The output is an [Nx6] array where the N rows are each point, and the 6 columns are x, y, z, r, g, b.

reconstruct(left_image: numpy.ndarray, left_camera_matrix: numpy.ndarray, right_image: numpy.ndarray, right_camera_matrix: numpy.ndarray, left_to_right_rmat: numpy.ndarray, left_to_right_tvec: numpy.ndarray, left_mask: numpy.ndarray = None)

A derived class must implement this.

Camera parameters are those obtained from OpenCV.

Parameters:

• left_image – left image, BGR
• left_camera_matrix – [3x3] camera matrix
• right_image – right image, BGR
• right_camera_matrix – [3x3] camera matrix
• left_to_right_rmat – [3x3] rotation matrix
• left_to_right_tvec – [3x1] translation vector
• left_mask – mask image, single channel, same size as left_image

Returns:

[Nx6] point cloud in left camera space, where N is the number

of points, and 6 columns are x,y,z,r,g,b.

Video Segmentation

Base class (pure virtual interface) for classes to do video segmentation

class sksurgerysurfacematch.interfaces.video_segmentor.VideoSegmentor

Bases: object

Base class for classes that can segment a video image into a binary mask. For example, a deep network that can produce a mask of background=0, foreground=255.

segment(image: numpy.ndarray)

A derived class must implement this.

Parameters:image – image, BGR Returns:image, same size as input, 1 channel, uchar, [0-255].

Processing Pipelines

Register Point Cloud To Stereo Images

Pipeline to register 3D point cloud to 2D stereo video

class sksurgerysurfacematch.pipelines.register_cloud_to_stereo_reconstruction.Register3DToStereoVideo(video_segmentor: sksurgerysurfacematch.interfaces.video_segmentor.VideoSegmentor, surface_reconstructor: sksurgerysurfacematch.interfaces.stereo_reconstructor.StereoReconstructor, rigid_registration: sksurgerysurfacematch.interfaces.rigid_registration.RigidRegistration, left_camera_matrix: numpy.ndarray, right_camera_matrix: numpy.ndarray, left_to_right_rmat: numpy.ndarray, left_to_right_tvec: numpy.ndarray, left_mask: numpy.ndarray = None, z_range: list = None, radius_removal: list = None, voxel_reduction: list = None)

Bases: object

Class for single-shot, registration of 3D point cloud to stereo video.

register(reference_cloud: numpy.ndarray, left_image: numpy.ndarray, right_image: numpy.ndarray, initial_ref2recon: numpy.ndarray = None) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray, numpy.ndarray]

Main method to do a single 3D cloud to 2D stereo video registration.

Camera calibration parameters are in OpenCV format.

Parameters:

• reference_cloud – [Nx3] points, each row, x,y,z, e.g. from CT/MR.
• left_image – undistorted, BGR image
• right_image – undistorted, BGR image
• initial_ref2recon – [4x4] of initial rigid transform.

Returns:

residual, [4x4] transform, of reference_cloud to left camera space, [Mx3] downsampled xyz points and [Mx6] reconstructed point cloud, as [x, y, z, r, g, b] rows.

Register Point Cloud To Mosaic

Pipeline to register 3D point cloud to mosaic’ed surface reconstruction.

class sksurgerysurfacematch.pipelines.register_cloud_to_stereo_mosaic.Register3DToMosaicedStereoVideo(video_segmentor: sksurgerysurfacematch.interfaces.video_segmentor.VideoSegmentor, surface_reconstructor: sksurgerysurfacematch.algorithms.reconstructor_with_rectified_images.StereoReconstructorWithRectifiedImages, rigid_registration: sksurgerysurfacematch.interfaces.rigid_registration.RigidRegistration, left_camera_matrix: numpy.ndarray, right_camera_matrix: numpy.ndarray, left_to_right_rmat: numpy.ndarray, left_to_right_tvec: numpy.ndarray, min_number_of_keypoints: int = 25, max_fre_threshold=2, left_mask: numpy.ndarray = None, z_range: list = None, radius_removal: list = None, voxel_reduction: list = None)

Bases: object

Class to register a point cloud to a series of surfaces derived from stereo video, and stitched together.

grab(left_image: numpy.ndarray, right_image: numpy.ndarray)

Call this repeatedly to grab a surface and use ORM key points to match previous reconstruction to the current frame.

Parameters:

• left_image – undistorted, BGR image
• right_image – undistorted, BGR image

register(point_cloud: numpy.ndarray, initial_transform: numpy.ndarray = None)

Registers a point cloud to the internal mosaicc’ed reconstruction.

Parameters:

• point_cloud – [Nx3] points, each row, x,y,z, e.g. from CT/MR.
• initial_transform – [4x4] of initial rigid transform.

Returns:

residual, [4x4] transform, of point_cloud to left camera space,

and [Mx6] reconstructed point cloud, as [x, y, z, r, g, b] rows.

reset()

Reset’s internal data members, so that you can start accumulating data again.