Joint registration and segmentation of images using deep learning

• US 10,726,555 B2
• Filed: 06/06/2018
• Issued: 07/28/2020
• Est. Priority Date: 06/06/2018
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Garnavi, Rahil, Ge, Zongyuan, Mahapatra, Dwarikanath, Sedai, Suman
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US16/001,566
• Publication Number:
  US 20190378274A1
• Time in Patent Office: 783 Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/086 using evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 2207/10008 from scanner, fax or copier
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/38 Registration of image sequences

Abstract

A system for registering and segmenting images includes an image scanner configured to acquire an image pair including a first image at a first time and a second image at a second time that is after the first time. A joint registration and segmentation server receives the image pair from the image scanner and simultaneously performs joint registration and segmentation on the image pair using a single deep learning framework. A computer vision processor receives an output of the joint registration and segmentation server and characterizes how a condition has progressed from the first time to the second time therefrom. A user terminal presents the characterization to a user.

Claims

1. A system for registering and segmenting images, comprising:

an image scanner configured to acquire an image pair including a first image at a first time and a second image at a second time that is after the first time;

a joint registration and segmentation server configured to receive the image pair from the image scanner and to simultaneously and concurrently joint registration and segmentation on the image pair as part of a single optimization problem using a single deep learning framework;

a computer vision processor configured to receive an output of the joint registration and segmentation server and to characterize how a condition has progressed from the first time to the second time therefrom; and

a user terminal configured to present the characterization to a user.

2. The system of claim 1, wherein the image scanner is a computed tomography (CT) scanner, a magnetic resonance imager (MRI), a camera, or a satellite-based imager.

3. The system of claim 1, wherein the first time and the second time are at different days and the first image and the second image are acquired in distinct imaging sessions.

4. The system of claim 1, wherein the joint registration and segmentation server is configured to perform joint registration and segmentation on the image pair using a neural network that has been trained, before the acquisition of the first and second images, using training data.

5. The system of claim 4, wherein the neural network includes generative adversarial networks (GANs) which may use one of the first and second images as a floating image and the other of the first and second images as a reference image.

6. The system of claim 1, wherein the output of the joint registration and segmentation server includes a segmentation mask and a registered image.

7. The system of claim 1, wherein the computer vision processor is configured to perform computer-aided diagnosis.

8. The system of claim 1, Wherein the user terminal includes a display device for displaying the characterization to the user.

9. A method for registering and segmenting images, comprising:

acquiring an image pair including a first image at a first time and a second image at a second time that is after the first time;

simultaneously and concurrently performing joint registration and segmentation on the image pair as part of a single optimization problem using a single deep learning framework; and

characterizing how a condition has progressed from the first time to the second time from a result of the joint registration and. segmentation of the image pair.

10. The method of claim 9, wherein the first and second images are computed tomography (CT) images, magnetic resonance imager (MRI) images, camera images, or satellite images.

11. The method of claim 9, wherein the first time and the second time are at different days and the first image and the second image are acquired in distinct imaging sessions.

12. The method of claim 9, wherein the joint registration and segmentation is performed on the image pair using a neural network that has been trained, before the acquisition of the first and second images, using training data.

13. The method of claim 12, wherein the neural network includes generative adversarial networks (GANs) which may use one of the first and second images as a floating image and the other of the first and second images as a reference image.

14. The method of claim 9, wherein the result of the joint registration and segmentation of the image pair includes a segmentation mask and a registered image.

15. The method of claim 9, wherein the characterizing of how the condition has progressed from the first time to the second time is part of a computer-aided diagnosis.

16. A computer program product for registering and segmenting images, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

simultaneously and concurrently performing, by the computer, joint registration and segmentation on an image pair as part of a single optimization problem using a single deep learning framework, the image pair including a first image at a first time and a second image at a second time that is after the first time; and

characterizing, by the computer, how a condition has progressed from the first time to the second time from a result of the joint registration and segmentation of the image pair.

17. The computer program product of claim 16, wherein the first and second images are computed tomography (CT) images, magnetic resonance imager (MRI) images, camera images, or satellite images.

18. The computer program product of claim 16, wherein the joint registration. and segmentation is performed on the image pair using a neural network that has been trained, before the acquisition of the first and second images, using training data.

19. The computer program product of claim 18, wherein the neural network includes generative adversarial networks (GANs) which may use one of the first and second images as a floating image and the other of the first and second images as a reference image.

20. The computer program product of claim 16, wherein the result of the joint registration and segmentation of the image pair includes a segmentation mask and a registered image.