Video object retrieval system and method

• US 9,201,958 B2
• Filed: 10/24/2013
• Issued: 12/01/2015
• Est. Priority Date: 10/24/2013
• Status: Active Grant

• Current Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Original Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Inventors: Wang, Haohong, Yang, Yimin
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US14/062,017
• Publication Number:
  US 20150117772A1
• Time in Patent Office: 768 Days
• Field of Search: 382/162, 382/165, 382/173, 382/305
• US Class Current: 1/1
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/7635 based on graphs, e.g. graph cuts or spectral clustering

Abstract

A video object retrieval method is provided. The method includes dividing an input video into a number of video shots and detecting target-class object occurrences and related-class object occurrences in each video shot. The method also includes pre-processing object bounding box images and performing automatic object extraction. Further, the method includes performing object-level feature extraction and integration and generating visual search results based on similarity scores.

Claims

1. A video object retrieval method, comprising:

dividing an input video into a number of video shots;

detecting target-class object occurrences and related-class object occurrences in each video shot;

pre-processing object bounding box images;

performing automatic object extraction via an automatic graph-cut algorithm;

performing object-level feature extraction and integration; and

generating visual search results based on similarity scores.

2. The method according to claim 1, wherein detecting target-class object occurrences and related-class object occurrences in each video shot further including:

acquiring multimodal cue information for all the detected object occurrences;

combining an appearance cue, a spatio-temporal cue, and a topological cue to aid object detection in a plurality of video shots; and

applying dynamic programming to find optimal object paths.

3. The method according to claim 1. wherein pre-processing object bounding box images further including:

performing histogram equalization to adjust global contrast of an image; and

carrying out image fusion to balance between image quality and global contrast level.

4. The method according to claim 1, wherein performing automatic object extraction further including:

initializing a trimap T with object rectangle consisting of a foreground, region T_F, a background region T_B and an uncertain pixel region T_U;

initializing background (α

_i=0) for iε

T_B and foreground (α

_i=1) for iε

T_U∪

T_F, where 1≦

i≦

N;

initializing the foreground and background Gaussian Mixture Models (GMMs) with k-means from sets α

_i=1 and α

_i=0, respectively;

assigning pixels to GMM components and learning GMM parameters from image data; and

estimating segmentation based on the graph-cut algorithm.

5. The method according to claim 1, wherein performing object-level feature extraction and integration further including:

performing object-level feature extraction for both auto color correlogram (ACC) and color and edge directivity descriptor (CEDD) features; and

carrying out similarity fusion.

6. The method according to claim 5, wherein:

provided that p and q are two feature vectors;

∥

p−

q∥

₁ is Manhattan distance between two feature vectors p and q, the ACC feature dissimilarity is defined by;

7. The method according to claim 5, wherein:

provided that p and q are two feature vectors;

p^T is the transpose of the vector p;

q^T is the transpose of the vector q, the CEDD feature dissimilarity is defined by;

8. The method according to claim 5, wherein:

provided that p and q are two feature vectors;

Sim_ACC(.) is the ACC feature similarity;

Sim_CEDD(.) is the CEDD feature similarity, a similarity score is defined by;

9. A video object retrieval system, comprising:

an object detection module configured to detect target-class object occurrences and related-class object occurrences in each video shot;

a preprocessing module configured to pre-process object bounding box images;

an object extraction module configured to perform automatic object extraction via an automatic graph-cut algorithm;

an object-level feature extraction module configured to perform object-level feature extraction;

a similarity computation module configured to compute auto color correlogram (ACC) feature similarity and color and edge directivity descriptor (CEDD) feature similarity; and

a similarity fusion and ranking module configured to carry out similarity fusion and generate final visual search results based on similarity scores.

10. The system according to claim 9, wherein the object detection module is further configured to:

acquire multimodal cue information for all the detected object occurrences;

combine an appearance cue, a spatio-temporal cue, and a topological cue to aid object detection in a plurality of video shots; and

apply dynamic programming to find optimal object paths.

11. The system according to claim 9, wherein the preprocessing module is further configured to:

perform histogram equalization to adjust global contrast of an image; and

carry out image fusion to balance between image quality and global contrast level.

12. The system according to claim 9, wherein the object extraction module is further configured to:

initialize a trimap T with object rectangle consisting of a foreground region T_F, a background region T_B and an uncertain pixel region T_U;

initialize background (α

_i=0) for iε

T_B and foreground (α

_i=1) for iε

T_U∪

T_F, where 1≦

i≦

N;

initialize the foreground and background Gaussian Mixture Models (GMMs) with k-means from sets α

_i=1 and α

_i=0, respectively;

assign pixels to GMM components and learn GMM parameters from image data; and

estimate segmentation based on the graph-cut algorithm.

13. The system according to claim 9, wherein the object-level feature extraction module is further configured to:

perform object-level feature extraction for both auto color correlogram (ACC) and color and edge directivity descriptor (CEDD) features.

14. The system according to claim 13, wherein:

provided that p and q are two feature vectors;

∥

p−

q∥

₁ is Manhattan distance between two feature vectors p and q, the ACC feature dissimilarity is defined by;

15. The system according to claim 13, wherein:

provided that p and q are two feature vectors;

p^T is the transpose of the vector p;

q_T is the transpose of the vector q, the CEDD feature dissimilarity is defined by;

16. The system according to claim 13, wherein:

provided that p and q are two feature vectors;

Sim_ACC(.) is the ACC feature similarity;

Sim_CEDD(.) is the CEDD feature similarity, a similarity score is defined by;