Hierarchical video search and recognition system

• US 8,874,584 B1
• Filed: 02/24/2010
• Issued: 10/28/2014
• Est. Priority Date: 02/24/2010
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Chen, Yang, Medasani, Swarup, Allen, David L., Jiang, Qin, Owechko, Yuri, Lu, Tsai-Ching
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): HAVAN, HUNG T
• Application Number:
  US12/660,320
• Time in Patent Office: 1,707 Days
• Field of Search: 707/724, 707/725
• US Class Current: 707/741
• CPC Class Codes: G06F 16/7854 using shape G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion

Abstract

Described is a system for content recognition, search, and retrieval in visual data. The system is configured to perform operations of receiving visual data as an input, processing the visual data, and extracting distinct activity-agnostic content descriptors from the visual data at each level of a hierarchical content descriptor module. The resulting content descriptors are then indexed with a hierarchical content indexing module, wherein each level of the content indexing module comprises a distinct set of indexed content descriptors. The visual data, generated content descriptors, and indexed content descriptors are then stored in a storage module. Finally, based on a content-based query by a user, the storage module is searched, and visual data containing the content of interest is retrieved and presented to the user. A method and computer program product for content recognition, search, and retrieval in visual data are also described.

Claims

1. A system for content recognition, search, and retrieval in visual data, the system comprising one or more processors configured to perform operations of:

receiving unlabeled visual data as an input;

extracting a set of activity-agnostic content descriptors from the unlabeled, visual data with a hierarchical content descriptor module having a plurality of levels, resulting in a set of generated activity-agnostic content descriptors, wherein each level of the hierarchical content descriptor module generates a distinct set of activity-agnostic content descriptors;

indexing the generated activity-agnostic content descriptors with a hierarchical content indexing module having a plural of levels corresponding to the plurality of levels of the hierarchical content descriptor module, resulting in a set of indexed content descriptors;

wherein each level of the content indexing module comprises a distinct set of indexed content descriptors from a corresponding level in the hierarchical content descriptor module;

storing the unlabeled visual data, the set of generated activity-agnostic content descriptors, and the set of indexed content descriptors in a storage module;

searching the storage module and retrieving visual data containing, a content of interest based on a user query;

refining the user query based on feature weighting; and

presenting the visual data containing the content of interest to the user.

2. The system for content recognition, search, and retrieval in visual data as set forth in claim 1, wherein the query is an unlabeled video clip containing a content of interest.

3. The system for content recognition, search, and retrieval in visual data as set forth in claim 2, wherein the system is further configured to compare a set of generated activity-agnostic content descriptors from the unlabeled video clip to the set of generated activity-agnostic content descriptors in the storage module to retrieve a match.

4. The system for content recognition, search, and retrieval in visual data as set forth in claim 3, wherein the hierarchical content descriptor module comprises a first level for detecting a set of salient spatiotemporal regions which are abstracted into a set of activity-agnostic spatiotemporal visual words, wherein a collection of each of the sets of activity-agnostic spatiotemporal visual words form a visual vocabulary.

5. The system for content recognition, search, and retrieval in visual data as set forth in claim 4, wherein the hierarchical content descriptor module comprises a second level for modeling a set of motion trajectories of at least one moving object or salient blob in the unlabeled visual data.

6. The system for content recognition, search, and retrieval in visual data as set forth in claim 5, wherein the hierarchical content descriptor module comprises a third level which employs a set of graphical activity representations.

7. A computer-implemented method for content recognition, search, and retrieval in visual data, the method comprising an act of causing a processor to perform operations of:

receiving unlabeled visual data as an input;

extracting a set of activity-agnostic content descriptors from the unlabeled visual data with a hierarchical content descriptor module having a plurality of levels, resulting in a set of generated activity-agnostic content descriptors, wherein each level of the hierarchical content descriptor module generates a distinct set of activity-agnostic content descriptors;

indexing the generated activity-agnostic content descriptors with a hierarchical content indexing module having a plurality of levels corresponding to the plurality of levels of the hierarchical content descriptor module, resulting in a set of indexed content descriptors;

wherein each level of the content indexing module comprises a distinct set of indexed content descriptors from a corresponding level in the hierarchical content descriptor module;

storing the unlabeled visual data, the set of generated activity-agnostic content descriptors, and the set of indexed content descriptors in a storage module;

searching the storage module and retrieving visual data containing a content of interest based on a user query;

refining the user query based on feature weighting; and

presenting the visual data containing the content of interest to the user.

8. The method for content recognition, search, and retrieval in visual data as set forth in claim 7, wherein the query is an unlabeled video clip containing a content of interest.

9. The method for content recognition, search, and retrieval in visual data as set forth in claim 8, further comprising an act of comparing a set of generated activity-agnostic content descriptors from the unlabeled video clip to the set of generated activity-agnostic content descriptors in the storage module to retrieve a match.

10. The method for content recognition, search, and retrieval in visual data as set forth in claim 9, wherein the hierarchical content descriptor module comprises a first level for detecting a set of salient spatiotemporal regions which are abstracted into a set of activity-agnostic spatiotemporal visual words, wherein a collection of each of the sets of activity-agnostic spatiotemporal visual words form a visual vocabulary.

11. The method for content recognition, search, and retrieval in visual data as set forth in claim 10, wherein the hierarchical content descriptor module comprises a second level for modeling a set of motion trajectories of at least one moving object or salient blob in the unlabeled visual data.

12. The method for content recognition, search, and retrieval in visual data as set forth in claim 11, wherein the hierarchical content descriptor module comprises a third level which employs a set of graphical activity representations.

13. A computer program product for content recognition, search, and retrieval in visual data, the computer program product comprising computer-readable instruction means stored on a computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving unlabeled visual data as an input;

extracting a set of activity-agnostic content descriptors from the unlabeled visual data with a hierarchical content descriptor module having a plurality of levels, resulting in a set of generated activity-agnostic content descriptors, wherein each level of the hierarchical content descriptor module generates a distinct set of activity-agnostic content descriptors;

indexing the generated activity-agnostic content descriptors with a hierarchical content indexing module having a plurality of levels corresponding to the plurality of levels of the hierarchical content descriptor module, resulting in a set of indexed content descriptors;

wherein each level of the content indexing module comprises a distinct set of indexed content descriptors from a corresponding level in the hierarchical content descriptor module;

storing the unlabeled visual data, the set of generated activity-agnostic content descriptors, and the set of indexed content descriptors in a storage module;

searching the storage module and retrieving visual data containing a content of interest based on a user query;

refining the user query based on feature weighting; and

presenting the visual, data containing the content of interest to the user.

14. The computer program product for content recognition, search and retrieval in visual data as set forth in claim 13, wherein the query is an unlabeled video clip containing a content of interest.

15. The computer program product for content recognition, search, and retrieval in visual data as set forth in claim 14, further comprising instruction means for comparing a set of generated activity-agnostic, content descriptors from the unlabeled video clip to the set of generated content activity-agnostic descriptors in the storage module to retrieve a match.

16. The computer program product for content recognition, search, and retrieval in visual data as set forth in claim 15, wherein the hierarchical content descriptor module comprises a first level for detecting, a set of salient spatiotemporal regions which are abstracted into a set of activity-agnostic spatiotemporal visual words, wherein a collection of each of the sets of the activity-agnostic spatiotemporal visual words form a visual vocabulary.

17. The computer program product for content recognition, search, and retrieval in visual data as set forth in claim 16, wherein the hierarchical content descriptor module comprises a second level for modeling a set of motion trajectories of at least one moving object or salient blob in the unlabeled visual data.

18. The computer program product for content recognition, search, and retrieval in visual data as set forth in claim 17, wherein the hierarchical content descriptor module comprises a third level which employs a set of graphical activity representations.