Deformable registration of magnetic resonance and ultrasound images using biomechanical models

• US 10,621,720 B2
• Filed: 04/27/2017
• Issued: 04/14/2020
• Est. Priority Date: 04/27/2017
• Status: Active Grant

• Current Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Pheiffer, Thomas, Kapoor, Ankur, Park, Jin-hyeong, Kamen, Ali
• Primary Examiner(s): Bella, Matthew C
• Assistant Examiner(s): Shin, Brian D
• Application Number:
  US15/499,302
• Publication Number:
  US 20180315181A1
• Time in Patent Office: 1,083 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/10136 3D ultrasound image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30081 Prostate
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

A computer-implemented method for performing deformable registration between Magnetic Resonance (MR) and Ultrasound (US) images include receiving an MR volume depicting an organ and segmenting the organ from the MR volume to yield a first 3D point representation of the organ in MR coordinates. Additionally, a US volume depicting an organ is received and the organ is segmented from the US volume to yield a second 3D point representation of the organ in US coordinates. Next, a plurality of point correspondences between the first 3D point representation and the second 3D point representation are determined. Then, a biomechanical model is applied to register the MR volume to the US volume. The plurality of point correspondences are used as displacement boundary conditions for the biomechanical model.

Claims

1. A computer-implemented method for performing deformable registration between Magnetic Resonance (MR) and Ultrasound (US) images, the method comprising:

receiving an MR volume depicting an organ;

segmenting the organ from the MR volume to yield a first 3D point representation of the organ in MR coordinates, wherein segmentation of the organ from the MR volume is automatically performed using one or more machine learning models to detect a MR bounding box containing the organ in the MR volume and computing the first 3D point representation by applying one or more shape models to portions of the MR volume within the MR bounding box;

receiving a US volume depicting the organ;

segmenting the organ from the US volume to yield a second 3D point representation of the organ in US coordinates, wherein segmentation of the organ from the US volume is automatically performed using the one or more machine learning models to detect a US bounding box containing the organ in the US volume and computing the second 3D point representation by applying the one or more shape models to portions of the US volume within the US bounding box;

determining a plurality of point correspondences between the first 3D point representation and the second 3D point representation; and

applying a biomechanical model to register the MR volume to the US volume, wherein the plurality of point correspondences are used as displacement boundary conditions for the biomechanical model.

2. The method of claim 1, wherein the MR volume and the US volume are both acquired during an intraoperative procedure.

3. The method of claim 1, wherein the first 3D point representation and the second 3D point representation are each a mesh.

4. The method of claim 1, wherein the first 3D point representation and the second 3D point representation are each a point cloud.

5. The method of claim 1, wherein (i) the segmentation of the MR volume is performed prior to the segmentation of the US volume and (ii) size and shape data from the segmentation of the MR volume is used as a constraint for the segmentation of the US volume.

6. The method of claim 1, wherein the one or more machine learning models are trained using training dataset comprising a plurality of training MR volumes and a plurality of training US volumes and the training process comprises:

for each volume included in the training dataset, creating annotation data using a mesh representation of the respective volume defined by a set of points and triangular information for faces of the mesh representation;

defining a bounding box around each volume in the training dataset, wherein the bounding box is defined by three position values, three orientations values, and three size values;

computing an organ shape within each bounding box; and

using the organ shape and the annotation data for each volume in the training dataset to train the one or more machine learning models.

7. The method of claim 6, wherein (i) the training process is applied to the plurality of training MR volumes before the training process is applied to the plurality of US volumes and (ii) the three size values used in defining the bounding box for each US volume are defined using the three size values for the bounding box of a corresponding MR volume in the training dataset.

8. The method of claim 1, further comprising:

segmenting subsurface structures of the organ from subsurface imaging data to yield a third 3D point representation of the organ,

wherein the plurality of point correspondences further comprises point correspondences (i) between the first 3D point representation and the third 3D point representation and (ii) between the second 3D point representation and the third 3D point representation.

9. The method of claim 1, further comprising:

warp subsurface image data in the MR volume into US coordinate space based on the registration of the MR volume to the US volume; and

displaying the subsurface image data overlaid on the US volume.

10. A computer-implemented method for performing deformable registration between images in two image modalities, the method comprising:

segmenting an organ from a first image volume acquired in a first modality to yield a first 3D point representation of the organ, wherein segmentation of the organ from the first image volume is automatically performed using one or more machine learning models to detect a first bounding box containing the organ in the first image volume and computing the first 3D point representation by applying one or more shape models to portions of the first image volume within the first bounding box;

segmenting the organ from a second image volume acquired in a second modality to yield a second 3D point representation of the organ, wherein segmentation of the organ from the second image volume is automatically performed using the one or more machine learning models to detect a second bounding box containing the organ in the second image volume and computing the second 3D point representation by applying the one or more shape models to portions of the second image volume within the second bounding box;

determining a plurality of point correspondences between the first 3D point representation and the second 3D point representation; and

applying a biomechanical model to register the first image volume to the second image volume, wherein the plurality of point correspondences are used as displacement boundary conditions for the biomechanical model.

11. The method of claim 10, wherein the first 3D point representation and the second 3D point representation are each a mesh.

12. The method of claim 10, wherein the first 3D point representation and the second 3D point representation are each a point cloud.

13. The method of claim 10, wherein (i) the segmentation of the first image volume is performed prior to the segmentation of the second image volume and (ii) size and shape data from the segmentation of the first image volume is used as a constraint for the segmentation of the second image volume.

14. The method of claim 10, wherein the one or more machine learning models are trained using training datasets comprising a plurality of training first image volumes and a plurality of training second image volumes and the training process comprises:

for each volume included in the training dataset, creating annotation data using a mesh representation of the respective volume defined by a set of points and triangular information for faces of the mesh representation;

defining a bounding box around each volume in the training dataset, wherein the bounding box is defined by three position values, three orientations values, and three size values;

computing an organ shape within each bounding box; and

using the organ shape and the annotation data for each volume in the training dataset to train the one or more machine learning models.

15. The method of claim 14, wherein (i) the training process is applied to the plurality of training first image volumes before the training process is applied to the plurality of second training image volumes and (ii) the three size values used in defining the bounding box for each training second image volume are defined using the three size values for the bounding box of a corresponding first image volume in the training dataset.

16. The method of claim 10, further comprising:

segmenting subsurface structures of the organ from subsurface imaging data to yield a third 3D point representation of the organ,

wherein the plurality of point correspondences further comprises point correspondences (i) between the first 3D point representation and the third 3D point representation and (ii) between the second 3D point representation and the third 3D point representation.

17. The method of claim 10, further comprising:

warping subsurface image data in the first image volume based on the registration of the first image volume to the second image volume; and

displaying the subsurface image data overlaid on the second image volume.

18. A system performing deformable registration between Magnetic Resonance (MR) and Ultrasound (US) images, the system comprising:

a parallel computing platform comprising a plurality of processors configured to;

apply a first machine learning model to segment an organ from a MR volume to yield a first 3D point representation of the organ in MR coordinates, wherein the first machine learning model is used to detect a MR bounding box containing the organ in the MR volume and compute the first 3D point representation within the MR bounding box;

apply a second machine learning model to segment the organ from a US volume to yield a second 3D point representation of the organ in US coordinates, wherein the second machine learning model is used to detect a US bounding box containing the organ in the US volume and compute the second 3D point representation within the US bounding box;

determine a plurality of point correspondences between the first 3D point representation and the second 3D point representation; and

apply a biomechanical model to register the MR volume to the US volume, wherein the plurality of point correspondences is used as displacement boundary conditions for the biomechanical model.