Method and system for sharing-oriented personalized route planning via a customizable multimedia approach

• US 10,217,224 B2
• Filed: 12/22/2016
• Issued: 02/26/2019
• Est. Priority Date: 12/22/2016
• Status: Active Grant

• Current Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Original Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Inventors: Liao, Xinpeng, Ren, Xiaobo, Wang, Haohong
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US15/388,852
• Publication Number:
  US 20180182105A1
• Time in Patent Office: 796 Days
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

In accordance with some embodiments of the disclosed subject matter, a method and a system for recommending and sharing customized multimedia route planning are provided. The method includes: receiving a query image from a user, the query image containing an object-of-interest of the user, performing an integrative segmentation process to determine one or more contours of the object-of-interest in the query image; determining a route having a maximum area overlap with the one or more contours of object-of-interest on a map image; generating an output image including the object-of-interest and the route; and recommending the output image to the user, and sharing the output image on a social network platform.

Claims

1. A method for recommending and sharing customized multimedia route planning, comprising:

receiving a query image from a user, the query image containing an object-of-interest of the user;

performing an integrative segmentation process to determine one or more contours of the object-of-interest in the query image;

determining a route having a maximum area overlap with the one or more contours of object-of-interest on a map image;

generating an output image including the object-of-interest and the route; and

recommending the output image to the user, and sharing the output image on a social network platform.

2. The method of claim 1, further comprising:

receiving additional user specified information with the query image;

wherein the additional user specified information includes an object bounding box, a user geographical location, a map ratio, and an expected route length.

3. The method of claim 2, wherein:

the map image is obtained from an online map application based on the user geographical location and the map ratio; and

the route is determined based on the one or more contours of the object-of-interest in the query image, the map image, and the expected route length.

4. The method of claim 2, wherein the integrative segmentation process comprises:

performing an initialization of the query image to obtain an initial trimap;

performing a first automatic segmentation process to identify an approximate border of the object-of-interest from the initial trimap;

performing a user interaction process to edit a segmentation result of the first automatic segmentation process to generate an updated trimap; and

performing a second automatic segmentation process to identify an accurate border of the object-of-interest from the updated trimap.

5. The method of claim 4, wherein performing an initialization of the query image comprises:

determining an initial foreground region, an initial background region, and an initial remaining region of the query image based on the object bounding box; and

setting Gaussian Mixture Models to each pixel in the initial foreground region and the initial background region.

6. The method of claim 4, wherein performing the first automatic segmentation process comprises:

assigning Gaussian Mixture Model components to each pixel in the initial remaining region;

learning Gaussian Mixture Model parameters from data of grey values of the initial trimap;

determining an estimate segmentation by using a minimum cut estimation algorithm to solve an equation of Gibbs energy, wherein the equation of Gibbs energy depends on the Gaussian Mixture Model components and the Gaussian Mixture Model parameters;

repeating the above three steps until a condition of convergence of iterative minimization is satisfied; and

performing a border matting to generate the approximate border of the object-of-interest from the initial trimap.

7. The method of claim 4, wherein performing the user interaction process comprises:

receiving user input on the segmentation result of the first automatic segmentation process, wherein the user input includes at least one of a background brush and a foreground brush;

fixing pixels corresponding to areas indicated by the background brush and the foreground brush respectively; and

updating the segmentation result of the query image accordingly to generate the updated trimap.

8. The method of claim 4, wherein performing the second automatic segmentation process comprises:

determining an updated segmentation of the update trimap by using a minimum cut estimation algorithm; and

performing a refine operation to identify the accurate border of the object-of-interest from the updated trimap.

9. The method of claim 8, wherein the refine operation comprises:

assigning Gaussian Mixture Model components to each pixel in an updated remaining region of the update trimap;

learning Gaussian Mixture Model parameters from data of grey values of the updated trimap;

determining an estimate segmentation by using a minimum cut estimation algorithm to solve an equation of Gibbs energy, wherein the equation of Gibbs energy depends on the Gaussian Mixture Model components and the Gaussian Mixture Model parameters;

repeating the above three steps until a condition of convergence of iterative minimization is satisfied; and

performing a border matting to generate the accurate border of the object-of-interest from the updated trimap;

wherein the accurate border includes the one or more contours of object-of-interest.

10. The method of claim 1, wherein determining the route having the maximum area overlap with the one or more contours of object-of-interest on the map image comprises:

employing a MorphSnakes algorithm to search a route candidate on the map image in eight different orientations;

using a morphological Geodesic Active Contour framework to evolve the route candidate by minimizing an energy functional; and

performing an evaluation method using distortion error to determine the route having the maximum area overlap with the one or more contours of object-of-interest on the map image.

11. The method of claim 10, wherein the energy functional is represented as:

E(C)=∫

₀^length(C)g(I)(C(s))ds=∫

₀¹g(I)(C(p)·

|C_p|)dp,

wherein ds=|C_p|dp is an Euclidean arc-length parametrization of a curve C; and

g(I) is a mapping;

^d→

⁺, x→

g(I)(x), which allows selecting regions-of-interest in the image I.

12. The method of claim 10, wherein employing the MorphSnakes algorithm comprises:

approximating numerical solutions of partial differential equations of curve evolution of the route candidate by successively applying a set of curvature morphological operators;

wherein the set of curvature morphological operators are defined on a binary level-set function.

13. The method of claim 12, further comprising:

introducing a norm metric between two successive level sets of the binary level-set function to assess a convergence of the curve evolution of the route candidate.

14. A system for recommending and sharing customized multimedia route planning, comprising:

one or more hardware processors, and

a memory, wherein the one or more hardware processors are configured to;

receive a query image from a user, the query image containing an object-of-interest of the user;

perform an integrative segmentation process to determine one or more contours of the object-of-interest in the query image;

determine a route having a maximum area overlap with the one or more contours of object-of-interest on a map image;

generate an output image including the object-of-interest and the route; and

recommend the output image to the user, and share the output image on a social network platform.

15. The system of claim 14, wherein the one or more hardware processors are further configured to:

receive additional user specified information with the query image;

wherein;

the additional user specified information includes an object bounding box, a user geographical location, a map ratio, and an expected route length;

the map image is obtained from an online map application based on the user geographical location and the map ratio; and

the route is determined based on the one or more contours of an object-of-interest in the query image, the map image, and the expected route length.

16. The system of claim 15, wherein the one or more hardware processors are further configured to:

perform an initialization of the query image to obtain an initial trimap;

perform a first automatic segmentation process to identify an approximate border of the object-of-interest from the initial trimap; and

perform a user interaction process to edit a segmentation result of the first automatic segmentation process to generate an updated trimap;

wherein a second automatic segmentation process is further performed to identify an accurate border of the object-of-interest from the updated trimap.

17. The system of claim 16, wherein the one or more hardware processors are further configured to:

determine a foreground region, a background region, and a remaining region of the query image based on the object bounding box; and

set Gaussian Mixture Models to each pixel in the foreground region and the background region.

18. The system of claim 16, wherein the one or more hardware processors are further configured to:

assign Gaussian Mixture Model components to each pixel in the remaining region;

learn Gaussian Mixture Model parameters from data of grey values of a trimap;

determine an estimate segmentation by using a minimum cut estimation algorithm to solve an equation of Gibbs energy, wherein the equation of Gibbs energy depends on the Gaussian Mixture Model components and the Gaussian Mixture Model parameters;

repeat the above three steps until a condition of convergence of iterative minimization is satisfied; and

perform a border matting to generate a border of the object-of-interest from the trimap.

19. The system of claim 16, wherein the one or more hardware processors are further configured to:

receive user input on the segmentation result of the first automatic segmentation process, wherein the user input includes at least one of a background brush and a foreground brush;

fix pixels corresponding to areas indicated by the background brush and the foreground brush respectively; and

update the segmentation result of the query image accordingly to generate the updated trimap.

20. The system of claim 15, wherein the one or more hardware processors are further configured to:

employ a MorphSnakes algorithm to search a route candidate on the map image in eight different orientations;

use a morphological Geodesic Active Contour framework to evolve the route candidate by minimizing an energy functional including;

approximate numerical solutions of partial differential equations of curve evolution of the route candidate by successively applying a set of curvature morphological operators, wherein the set of curvature morphological operators are defined on a binary level-set function, and

introduce a norm metric between two successive level sets of the binary level-set function to assess a convergence of the curve evolution of the route candidate; and

perform an evaluation method using distortion error to determine the route having the maximum area overlap with the one or more contours of object-of-interest on the map image.