Method for Semantically Labeling an Image of a Scene using Recursive Context Propagation

• US 20160055237A1
• Filed: 08/20/2014
• Published: 02/25/2016
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Tuzel, Oncel, Liu, Ming-Yu, Sharma, Abhishek
• Granted Patent: US 9,558,268 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/759 Region-based matching
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A method semantically labels an image acquired of a scene by first obtaining a local semantic feature for each local region in the image. The local semantic features are combined recursively to form intermediate segments until a semantic feature for the entire image is obtained. Then, the semantic feature for the entire image is decombined recursively into intermediate segments until an enhanced semantic feature for each local region is obtained. Then, each local region is labeled according to the enhanced semantic feature.

Claims

1. A method for semantically labeling an image acquired of a scene, comprising steps of:

obtaining a local semantic feature from each local region in the image;

combining recursively the local semantic features to form intermediate segments until a semantic feature for the entire image is obtained;

decombining recursively the semantic feature for the entire image into intermediate segments until an enhanced semantic feature for each local region is obtained; and

labeling each local region according to the enhanced semantic feature, wherein the steps are performed in a processor.

2. The method of claim 1, wherein the local regions are superpixels.

3. The method of claim 1, wherein the local regions include one or more pixels.

4. The method of claim 1, wherein the local features are extracted from local regions according to a multi scale convolutional neural network.

5. The method of claim 1, wherein the combining and the decombining are according to a parse tree of the image.

6. The method of claim 1, further comprising:

synthesizing randomly balanced binary parse trees of nodes for the combining and the decombining.

7. The method of claim 1,

where the mapping is according to x_i=F_sem(v_i;

θ

_sem),

wherein x_i is the semantic feature, v_i is the local feature, and θ

_sem is a semantic mapping parameter,

where the combing is according to x_i,j=F_sem([x_i,x_j];

θ

_com),

wherein x_i,j a semantic feature of a parent node obtained from child nodes x_i,x_j in a binary parse tree, and θ

_com is a combining parameter;

where the decombining is according to {tilde over (x)}_i=F_dec([{tilde over (x)}_i,j,x_i];

θ

_dec),

wherein {tilde over (x)} is an enhanced semantic feature of a child node obtained from a parent node {tilde over (x)}_i,j and x_i, and θ

_sem is a decombining parameter, and

wherein the labeling is according to y_i=F_lab({tilde over (x)}_i;

θ

_lab),

wherein y_i is a label, and θ

_sem is a labeling parameter.

8. The method of claim 6, and further comprising:

adding side information to the nodes.

9. The method of claim 8, wherein the side information encodes static knowledge about the nodes.

10. The method of claim 8, wherein the side information is an average of locations of the nodes and sizes of the nodes.

11. The method of claim 7, wherein the extracting, combining, decombining and labeling functions use neural network.

12. The method of claim 1, wherein the local semantic features are pixel color, gradient and texture features.

13. The method of claim 7, further comprising:

learning the parameters of the extracting, the combining, the decombining and the labeling using training data.

14. The method of claim 13, wherein the learning minimizes a difference between true labels and predicted labels of the pixels in the training data.

15. The method of claim 7, wherein function of the combining and the decombining are recursive.

16. The method of claim 5, wherein the parse tree is a hierarchical segmentation representation of the image of the scene.

17. The method of claim 5, wherein the parse tree is a hierarchical segmentation representation of a part of the image of the scene.

18. The method of claim 1, wherein the semantic feature for the entire image is used to classify the image according to scene categories.

19. The method of claim 1, wherein the semantic feature for the entire image is used for clustering collections of images.