Method and apparatus for image processing

• US 9,799,098 B2
• Filed: 04/24/2008
• Issued: 10/24/2017
• Est. Priority Date: 04/24/2007
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology, Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Original Assignee: Massachusetts Institute of Technology, Max-Planck-Gesellschaft Zur Foerderung Der Wissenschaften e.V. (Max-Planck-Gesellschaft zur Frderung der Wissenschaften e)
• Inventors: Seung, H. Sebastian, Murray, Joseph F., Jain, Viren, Turaga, Srinivas C., Helmstaedter, Moritz, Denk, Winfried
• Primary Examiner(s): Roz, Mark
• Application Number:
  US12/597,406
• Publication Number:
  US 20100183217A1
• Time in Patent Office: 3,470 Days
• Field of Search: 382260, 382180, 382298-300
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]

Abstract

Identifying objects in images is a difficult problem, particularly in cases an original image is noisy or has areas narrow in color or grayscale gradient. A technique employing a convolutional network has been identified to identify objects in such images in an automated and rapid manner. One example embodiment trains a convolutional network including multiple layers of filters. The filters are trained by learning and are arranged in successive layers and produce images having at least a same resolution as an original image. The filters are trained as a function of the original image or a desired image labeling; the image labels of objects identified in the original image are reported and may be used for segmentation. The technique can be applied to images of neural circuitry or electron microscopy, for example. The same technique can also be applied to correction of photographs or videos.

Claims

1. A method for image processing comprising:

training a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges.

2. The method of claim 1 wherein training the convolutional network includes training with back propagation learning.

3. The method of claim 1 wherein training the convolutional network includes training with stochastic gradient learning.

4. The method of claim 1 further including training the convolutional network as a function of training data.

5. The method of claim 1 further including producing an image at a resolution higher than the original image.

6. The method of claim 1 further including configuring the convolutional network to alternate between applying linear filtering and non-linear transformations.

33. The method of claim 1 further comprising obviating subsampling in the convolutional network.

7. An image processing system comprising:

a memory; and

a processor, the processor being configured to execute;

a training module, stored in the memory, to train a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training module including a partitioning module configured to partition an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments and by cutting edges between nodes on boundaries of the edges.

8. The image processing system of claim 7 wherein the training module is configured to train the convolutional network with back propagation learning.

9. The image processing system of claim 7 wherein the training module is configured to train the convolutional network with stochastic gradient learning.

10. The image processing system of claim 7 wherein the training module is configured to train the convolutional network as a function of training data.

11. The image processing system of claim 7 wherein the convolutional network is arranged to produce an image at a resolution higher than the original image.

12. The image processing system of claim 7 wherein the convolutional network is configured to alternate between applying linear filtering and non-linear transformations.

13. A method for image processing comprising:

training a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters trained by learning and arranged in successive layers of filters to produce images in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges;

training the filters as a function of the original image or a desired image labeling; and

reporting image labels of objects identified in the original image.

14. The method of claim 13 wherein training the filters includes training with back propagation.

15. The method of claim 13 wherein training the filters includes training with stochastic gradient learning.

16. The method of claim 13 further including training the filters as a function of training data.

17. The method of claim 13 further including producing an image with a higher resolution than the original image.

18. The method of claim 13 further including configuring the convolutional network to alternate between applying linear filtering and non-linear transformations.

19. The method of claim 13 further including accepting imaging labels, grouping together neighboring pixels with similar labels to produce new image labels corresponding to objects, and reporting the new image labels corresponding to the objects.

20. The method of claim 19 further including displaying the objects on a screen with different illustrations representing differences between or among the objects.

21. The method of claim 20 further including applying different colors to the objects.

22. An image processing system comprising:

a processor, the processor being configured to execute;

a first training module to train a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters trained by learning and arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges;

a second training module to train the filters as a function of the original image or a desired image labeling; and

a reporting module to report image labels of objects identified in the original image.

23. The image processing system of claim 22 wherein the first training module is configured to train the convolutional network by back propagation.

24. The image processing system of claim 22 wherein the first training module is configured to train the convolutional network by gradient learning.

25. The image processing system of claim 22 wherein the second training module is configured to train the filters as a function of training data.

26. The image processing system of claim 22 wherein the convolutional network is arranged to produce an image with a higher resolution than the original image.

27. The image processing system of claim 22 wherein the convolutional network is arranged to alternate between applying linear filtering and non-linear transformations.

28. The image processing system of claim 22 further including a connected components module to accept imaging labels, group together neighboring pixels with similar labels to produce new image labels corresponding to objects, and report the new image labels corresponding to the objects.

29. The image processing system of claim 28 further including a display module to display the objects on a screen with different indications representing differences between or among the objects.

30. The image processing system of claim 29 further including a coloring module to apply the different indications as different colors to the objects.

31. A non-transitory computer-readable medium with computer instructions stored thereon, the computer instructions when executed by a processor cause an apparatus to:

train a convolutional network including defined by nodes and edges, between nearest neighbor nodes, that form a lattice of layers of filters arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges.

32. A non-transitory computer-readable medium with computer instructions stored thereon, the computer instructions when executed by a processor cause an apparatus to:

train a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters trained by learning and arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges;

train the filters as a function of the original image or a desired image labeling; and

report image labels of objects identified in the original image.