Method and system for building personalized knowledge base of semantic image segmentation via a selective random field approach

• US 9,939,272 B1
• Filed: 01/06/2017
• Issued: 04/10/2018
• Est. Priority Date: 01/06/2017
• Status: Active Grant

• Current Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Original Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Inventors: Liao, Xinpeng, Ren, Xiaobo, Wang, Haohong
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US15/400,233
• Time in Patent Office: 459 Days
• Field of Search: 382156, 382157, 382159, 382165, 382199, 382266, 382282, 382291, 128925, 706 15, 706 25
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 50/40 Business processes related to the transportation industry shipping G06Q10/083
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20096 Interactive definition of curve of interest
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

The present invention discloses a method for recommending an image-based route. The method includes: obtaining a query image from a user; performing an interactive segmentation process to determine one or more contours of an object-of-interest in the query image; determining a route having the maximum area overlap with the one or more contours of the object-of-interest on a map image; generating an output image including the object-of-interest and the route having the maximum area overlap with the contours of the object-of-interest; and recommending the output image to the user, and sharing the output image on a social network platform.

Claims

1. A method for recommending an image-based route, comprising:

obtaining a query image from a user;

performing an interactive segmentation process to determine one or more contours of an object-of-interest in the query image;

determining a route having a maximum area overlap with the one or more contours of the object-of-interest on a map image;

generating an output image including the object-of-interest and the route having the maximum area overlap with the contours of the object-of-interest; and

recommending the output image with the route to the user, and sharing the output image on a social network platform.

2. The method of claim 1, wherein performing an interactive segmentation process to determine one or more contours of an object-of-interest in the query image further includes:

feeding the query image through a fully connected convolutional network and a convolutional neural network message estimator for inferencing selective hidden-unit random field based on a personalized knowledge base;

continuously back-propagating error derivatives with updated ground truth triggered by unsatisfied segmentation results of unseen images; and

refining semantic segmentation and learning super-pixel selection criterion to build the personalized knowledge base.

3. The method of claim 2, wherein:

given a set of observations x and the parameters θ

, a conditional probability of the segmentation label s is modeled as;

4. A system for recommending an image-based route, comprising:

an interactive segmentation module configured to obtain a query image from a user and perform an interactive segmentation process to determine on or more contours of an object-of-interest in the query image;

a route searching module configured to determine a route having a maximum area overlap with the one or more contours of the object-of-interest on a map image; and

a route recommending module configured to generate an output image including the object-of-interest and the route having the maximum area overlap with the contours of the object-of-interest, and to recommend the output image with the route to the user to be shared on a social network platform.

5. The system of claim 4, wherein the interactive segmentation module includes:

an image selection unit configured to allow the user to select a query image;

an image editing unit configured to allow the user to edit a super-pixel contour overlapping the query image by providing the foreground and background constraints;

an image processing unit configured to process the query image to generate one or more super-pixel contours through semantic image segmentation; and

a personalized knowledge base unit configured to adaptively learn the user preferences or selection criteria with contextual cues in unseen settings to reduce the image editing effort by the user and to replace the user interactions eventually.

6. The system of claim 5, wherein:

the image processing unit also extracts and sends feedback from the user image editing to the personalized knowledge base unit; and

the semantic image segmentation is performed through a selective random field approach.

7. The system of claim 6, wherein:

the image processing unit further includes a fully convolutional network and a super-pixel selector.

8. The system of claim 5, wherein:

the route searching module searches a route on Google map with a MorphSnakes algorithm.

9. The system of claim 5, wherein:

the route recommending module recommends a route having the maximum overlap area with the super-pixel contour generated by the interactive segmentation module to the user.

10. A non-transitory computer-readable medium having computer program for, when being executed by a processor, performing a method for recommending an image-based route, the method comprising:

obtaining a query image from a user;

performing an interactive segmentation process to determine one or more contours of an object-of-interest in the query image;

determining a route having a maximum area overlap with the one or more contours of the object-of-interest on a map image;

generating an output image including the object-of-interest and the route having the maximum area overlap with the contours of the object-of-interest; and

recommending the output image with the route to the user, and sharing the output image on a social network platform.

11. The non-transitory computer-readable medium of claim 10, wherein performing an interactive segmentation process to determine one or more contours of an object-of-interest in the query image further includes:

feeding the query image through a fully connected convolutional network and a convolutional neural network message estimator for inferencing selective hidden-unit random field based on a personalized knowledge base;

continuously back-propagating error derivatives with updated ground truth triggered by unsatisfied segmentation results of unseen images; and

refining semantic segmentation and learning super-pixel selection criterion to build the personalized knowledge base.

12. The non-transitory computer-readable medium of claim 11, wherein:

given a set of observations x and the parameters θ

, a conditional probability of the segmentation label s is modeled as;