METHOD AND SYSTEM FOR EXTRACTION AND ASSOCIATION OF OBJECT OF INTEREST IN VIDEO

• US 20130101209A1
• Filed: 12/14/2012
• Published: 04/25/2013
• Est. Priority Date: 10/29/2010
• Status: Abandoned Application

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.), Peking University
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.), Peking University
• Inventors: Tian, Yonghong, Li, Jia, Gao, Yunchao, Zhang, Jun, Yan, Jun, Yu, Haonan
• Application Number:
  US13/715,632
• Publication Number:
  US 20130101209A1
• Time in Patent Office: Days
• US Class Current: 382/164
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/248 by interactive preprocessing or interactive shape modelling, e.g. feature points assigned by a user
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59

Abstract

The present disclosure relates to an image and video processing method, and in particular, to a two-phase-interaction-based extraction and association method for an object of interest in a video. In the method, a user performs coarse positioning interaction by an interactive method which is not limited to a normal manner and has a low requirement for prior knowledge; based on this, a certain extraction algorithm which is fast and easy to implement is adopted to perform multi-parameter extraction on the object of interest. In the method, on the basis of mining video information fully and ensuring user preference, in a manner where the viewing of the user is not affected, associate value-added information with the object which the user is interested in, thereby meeting the user'"'"'s requirement for deeply knowing and further exploring an attention area.

Claims

1. A method for extracting an object of interest in a video, comprising:

generating an attention degree parameter according to a position of point obtained in a coarse positioning process, wherein the attention degree parameter indicates an attention degree of each area in a video frame;

identifying a foreground area according to the attention degree of each area in the video frame;

performing convex hull processing on the foreground area to obtain candidate objects of interest, and determining an optimal candidate object of interest according to a user reselection result; and

extracting a visual feature of the optimal candidate object of interest, obtaining an optimal image in an image feature base according to the visual feature, matching out value-added information which corresponds to the optimal image in a value-added information base, and presenting the matched value-added information to the user.

2. The method according to claim 1, wherein obtaining the position of point in the coarse positioning process comprises:

through mouse clicking, recording the position of point which corresponds to a user interaction position;

or,

through an infrared three-dimensional positioning apparatus, obtaining a user interaction coordinate in a three-dimensional space so as to obtain the position of point which corresponds to the interaction position of the user.

3. The method according to claim 1, wherein after generating the attention degree parameter according to the position of point obtained in the coarse positioning process, the method further comprises:

dividing the video frame into several areas, and mapping the attention degree parameter to each video area.

4. The method according to claim 3, wherein identifying the foreground area according to the attention degree of each area in the video frame comprises:

according to the attention degree parameter, performing statistics on a representative feature of a pixel point in an area which is relevant to the attention degree parameter in the video frame;

classifying all pixel points on the video frame according to their representative features and similarity of each statistical type; and

after each pixel point is classified, taking the video area with the highest attention degree as the foreground area.

5. The method according to claim 3, wherein the attention degree parameter acts as an assistant factor for establishing a statistical data structure, and a statistical object of the statistical data structure is the representative feature of a pixel point on the video frame.

6. The method according to claim 1, wherein the visual feature comprises at least one of the following:

a color feature;

performing statistics to form a color histogram of the optimal candidate object of interest in a given color space to obtain a color feature vector;

a structure feature;

through a key point extraction algorithm, obtaining a structure feature vector of the optimal candidate object of interest;

a texture feature;

extracting texture of the optimal candidate object of interest through Gabor transformation to obtain a texture feature vector; and

an outline feature;

through a trace transformation algorithm, extracting a line which forms the optimal candidate object of interest to obtain an outline feature vector.

7. The method according to claim 6, wherein the structure feature comprises calculating an obtained surface feature with high robustness for changes such as rotation, scale transformation, translation, noise adding, color, and brightness, through investigating a structure numerical relationship between local features of the image.

8. The method according to claim 1, wherein the obtaining the optimal image in the image feature base according to the visual feature comprises:

searching the image feature base, calculating similarity of each visual feature, and selecting an image with highest similarity as the optimal image.

9. The method according to claim 8, further comprising:

performing weighting on a similarity result obtained through calculating for each visual feature according to prior proportion, and selecting an image with an optimal weighting result as the optimal image.

10. An system for extracting an object of interest in a video, comprising:

a basic interaction module, configured to provide a position of point obtained according to a coarse positioning process;

an object of interest extraction module, configured to generate an attention degree parameter according to the position of point provided by the user in the coarse positioning process, wherein the attention degree parameter is used to indicate an attention degree of each area in a video frame, identify a foreground area according to the attention degree of each area in the video frame, and perform convex hull processing on the foreground area to obtain candidate objects of interest;

an extending interaction module, configured to determine an optimal candidate object of interest according to a user reselection result; and

a value-added information searching module, configured to extract a visual feature of the optimal candidate object of interest, obtain an optimal image in an image feature base according to the visual feature, match out value-added information which corresponds to the optimal image in a value-added information base, and present the matched value-added information to the user.

11. The system according to claim 10, wherein the object of interest extraction module comprises:

a parameter generating submodule, configured to generate the attention degree parameter according to the position of point obtained in the coarse positioning process;

a feature statistic submodule, configured to perform statistics on a representative feature of a pixel point in an area which is relevant to the attention degree parameter in the video frame according to the attention degree parameter;

a foreground identifying submodule, configured to classify all pixel points on the video frame according to their representative features and similarity of each statistical type, and after each pixel point is classified, take a video area with highest attention degree as the foreground area; and

an object extraction submodule, configured to extract objects of interest from the foreground area by using a convex hull algorithm.

12. The system according to claim 10, wherein the value-added information searching module comprises the following submodules:

a feature extraction submodule, configured to extract a visual feature to be matched of the optimal candidate object of interest;

a feature communication submodule, configured to pass a searching feature between a server end and a client end;

an image matching submodule, configured to search the image feature base, calculate similarity of each visual feature, and select an image with highest similarity as the optimal image;

a result obtaining submodule, configured to match out the value-added information which corresponds to the optimal image in the value-added information base; and

a value-added information communication submodule, configured to pass the value-added information between the server end and the client end.