Segmenting irregular shapes in images using deep region growing with an image pyramid

• US 10,643,092 B2
• Filed: 06/21/2018
• Issued: 05/05/2020
• Est. Priority Date: 06/21/2018
• Status: Active Grant

• Current Assignee: Guerbet
• Original Assignee: International Business Machines Corporation
• Inventors: Dufort, Paul
• Primary Examiner(s): Yang, Qian
• Application Number:
  US16/014,801
• Publication Number:
  US 20190392243A1
• Time in Patent Office: 684 Days
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30061 Lung
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A system for determining a region of interest in an image. The system includes a memory and an electronic processor. The electronic processor is configured to create an image pyramid for the image. The image pyramid includes a plurality of layers. For each layer of the image pyramid, the electronic processor is configured to initialize internal states of nodes of a spatial lattice and iteratively update the internal states of the nodes using spatially gated propagation. Each node in the spatial lattice represents a block of pixels in the image and is connected to a node representing a neighboring block of pixels in the image. The electronic processor is also configured to identify the region of interest based on the internal states of the nodes at a convergence of the spatial lattice having nodes representing values included in a first layer of the image pyramid.

Claims

1. A method for identifying an object of interest in a medical image, the method comprising:

creating an image pyramid for the medical image, wherein the image pyramid includes a plurality of layers, each layer includes a plurality of values, each value represents a block of one or more pixels in the medical image, and each successive layer includes fewer values than a most previous layer;

for each layer of the image pyramid;

initializing internal states of nodes of a spatial lattice, wherein each node in the spatial lattice represents a block of one or more pixels in the medical image and is connected to at least one node representing a neighboring block of one or more pixels in the medical image; and

iteratively updating, using a neural network, the internal states of the nodes in the spatial lattice using spatially gated propagation, wherein at each iteration each node updates its internal state based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, and a new value of the node; and

identifying the object of interest within the medical image based on the values of the nodes at a convergence of the spatial lattice having nodes representing the values included in a first layer of the image pyramid.

2. The method according to claim 1, wherein iteratively updating, using a neural network, the internal states of the nodes includes updating a value in a vector of values associated with the internal states of the nodes.

3. The method according to claim 1, the method further comprising

performing, at each iteration for each layer of the image pyramid, a first convolution involving a first concatenation of previous internal states of the nodes representing the values included in a layer of the image pyramid and the values included in the layer of the image pyramid, and

storing results of performing the first convolution.

4. The method according to claim 3, the method further comprising performing, at each iteration for each layer of the image pyramid, a second convolution involving a second concatenation of results of performing the first convolution for a current layer of the image pyramid, a layer of the image pyramid directly above the current layer of the image pyramid, and a layer of the image pyramid directly below the current layer of the image pyramid.

5. The method according to claim 4, wherein each node updates its internal state based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, and a new value of the node includes using a squashing function and results of performing the second convolution.

6. The method according to claim 1, wherein creating the image pyramid includes performing convolutions on each value representing a brightness of each block of one or more pixels in the medical image, wherein each convolution involving a reduction of dimensions of input medical image data produces values that are used to represent the medical image in a next layer of the image pyramid.

7. The method according to claim 1, wherein each value representing the medical image in the first layer of the image pyramid corresponds to a pixel in the medical image.

8. The method according to claim 7, wherein identifying the object of interest within the medical image based on the values of the nodes at a convergence of the spatial lattice having nodes representing the values included in a first layer of the image pyramid includes using a final layer of the neural network to calculate a probability that each pixel in the medical image is included in the object of interest based on a value included in each vector of values associated with a node representing the values included in a first layer of the image pyramid;

and

determining, for each pixel, if the calculated probability is above a predetermined threshold.

9. The method according to claim 1, wherein the neighboring node is one selected from a group consisting of a node that represents a block of one or more pixels that is directly above, directly below, to the right of, and to the left of a block of one or more pixels represented by the node.

10. The method according to claim 1, wherein representing the medical image with fewer values creates a medical image with a lower resolution.

11. A system for determining a region of interest in an image, the system comprising

a memory;

and

an electronic processor, connected to the memory and configured to;

create an image pyramid for the image, the image pyramid including a plurality of layers,

for each layer of the image pyramid,

initialize internal states of nodes of a spatial lattice, wherein each node represents a block of one or more pixels in the image and is connected to at least one node representing a neighboring block of one or more pixels in the image, and

iteratively update, using a neural network, the internal states of the nodes in the spatial lattice using spatially gated propagation; and

identify the region of interest within the image based on the internal states of the nodes at a convergence of the spatial lattice having nodes representing values included in a first layer of the image pyramid.

12. The system according to claim 11, wherein each successive layer of the plurality of layers included in the image pyramid represents the image at a lower resolution than an image represented in a most previous layer of the image pyramid.

13. The system according to claim 12, wherein the electronic processor is configured to represent the image at a lower resolution by representing the image with fewer values.

14. The system according to claim 13, wherein the electronic processor is configured to perform, at each iteration for each layer of the image pyramid, a first convolution involving a first concatenation of previous internal states of the nodes representing the values included in the layer of the image pyramid and the values included in the layer of the image pyramid, and store results of performing the first convolution.

15. The system according to claim 14, wherein the electronic processor is configured to perform, at each iteration for each layer of the image pyramid, a second convolution involving a second concatenation of results of performing the first convolution for a current layer of the image pyramid, a layer of the image pyramid directly above the current layer of the image pyramid, and a layer of the image pyramid directly below the current layer of the image pyramid.

16. The system according to claim 15, wherein the electronic processor is configured to update the internal state based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node by using a squashing function and results of performing the second convolution.

17. The system according to claim 12, wherein the electronic processor is further configured to perform, in the first iteration, convolutions on each value representing a brightness of each block of one or more pixels in the image, wherein each convolution involving a reduction of dimensions of input image data produces values that are used to represent the image in a next layer of the image pyramid.

18. The system according to claim 11, wherein the electronic processor is configured to update the internal states of the nodes by, at each iteration, deciding for each node whether to maintain a value of the node from a previous iteration, to set a value of the node to a value of a neighboring node from a previous iteration, or set a new value of the node.

19. The system according to claim 18, wherein the neighboring node is one selected from the group consisting of a node that represents a block of one or more pixels in the image that is directly above, directly below, to the right of, and to the left of the block of one or more pixels in the image represented by the node.

20. The system according to claim 11, wherein the electronic processor is configured to iteratively update, using a neural network, the internal states of the nodes by updating a value in a vector of values associated with the internal states of the nodes.

21. The system according to claim 11, wherein the electronic processor is configured to identify an object of interest within the image based on the values of the nodes at a convergence of the spatial lattice having nodes representing values included in a first layer of the image pyramid by using a final layer of the neural network to calculate a probability that each pixel in the image is included in the object of interest based on each vector associated with a node representing the values included in a first layer of the image pyramid, and

determining, for each pixel, if the calculated probability is above a predetermined threshold.

22. Non-transitory computer-readable medium storing instructions that, when executed with an electronic processor, perform a set of functions, the set of functions comprising:

creating an image pyramid for an image, wherein the image pyramid includes a plurality of layers, each layer includes a plurality of values, each value represents a block of one or more pixels in the image, and each successive layer includes fewer values than a most previous layer;

for each layer of the image pyramid;

initializing internal states of nodes of a spatial lattice, wherein each node represents a block of one or more pixels in the image and is connected to at least one node representing a neighboring block of one or more pixels in of the image; and

iteratively updating, using a neural network, the internal states of the nodes in the spatial lattice using spatially gated propagation, wherein at each iteration each node updates its internal state based on at least one selected from the group consisting of a value of the node from a previous iteration, a value of a neighboring node from the previous iteration, or a new value of the node; and

identifying an object of interest within the image based on the values of the nodes at a convergence of the spatial lattice having nodes representing the values included in a first layer of the image pyramid.

23. The non-transitory computer-readable medium according to claim 22, wherein iteratively updating, using a neural network, the internal states of the nodes includes updating a value in a vector of values associated with the internal states of the nodes.

24. The non-transitory computer-readable medium according to claim 22, wherein identifying an object of interest within the image based on the values of the nodes at a convergence of the spatial lattice having nodes representing the values included in a first layer of the image pyramid includes using a final layer in the neural network to calculate a probability that each pixel in the image is included in the object of interest based on the vector associated with a node representing the values included in a first layer of the image pyramid;

and

determining, for each pixel, if the calculated probability is above a predetermined threshold.