Video description system and method

• US 8,370,869 B2
• Filed: 06/06/2006
• Issued: 02/05/2013
• Est. Priority Date: 11/06/1998
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), International Business Machines Corporation
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Paek, Seungyup, Benitez, Ana, Chang, Shih-Fu, Puri, Atul, Huang, Qian, Li, Chung-Sheng, Smith, John R., Bergman, Lawrence
• Primary Examiner(s): SHEPARD, JUSTIN E
• Application Number:
  US11/448,114
• Publication Number:
  US 20070245400A1
• Time in Patent Office: 2,436 Days
• Field of Search: 715/500.1
• US Class Current: 725/19
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06V 10/422 for representing the structure of the pattern or shape of an object therefor
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Systems and methods for describing video content establish video description records which include an object set (24), an object hierarchy (26) and entity relation graphs (28). Video objects can include global objects, segment objects and local objects. The video objects are further defined by a number of features organized in classes, which in turn are further defined by a number of feature descriptors (36, 38, and 40). The relationships (44) between and among the objects in the object set (24) are defined by the object hierarchy (26) and entity relation graphs (28). The video description records provide a standard vehicle for describing the content and context of video information for subsequent access and processing by computer applications such as search engines, filters and archive systems.

Claims

1. A non-transitory computer readable media containing digital information with at least one description record representing video content embedded within corresponding video information, the at least one description record generated by the method comprising:

generating one or more video object descriptions from said video information using video object extraction processing;

generating one or more video object hierarchy descriptions from said generated video object descriptions using object hierarchy construction and extraction processing, each video object hierarchy description describing at least one relationship between at least two objects in the video information; and

generating one or more entity relation graph descriptions from said generated video object descriptions using entity relation graph generation processing, wherein said one or more generated entity relation graph descriptions describes non-hierarchical relationships between said objects associated with said one or more generated video object descriptions.

2. The computer readable media of claim 1, wherein said video object descriptions, said video object hierarchy descriptions, and said entity relation graph descriptions further comprise one or more feature descriptions.

3. The computer readable media of claim 2, wherein said one or more feature descriptions are selected from the group consisting of media features, visual features, temporal features and semantic features.

4. The computer readable media of claim 3, wherein said semantic features are further defined by at least one feature description selected from the group consisting of who, what object, what action, where, when, why and text annotation.

5. The computer readable media of claim 3, wherein said visual features are further defined by at least one feature description selected from the group consisting of color, texture, position, size, shape, motion, camera motion, editing effect and orientation.

6. The computer readable media of claim 3, wherein said media features are further defined by at least one feature description selected from the group consisting of file format, file size, color representation, resolution, data file location, author, creation, scalable layer and modality transcoding.

7. The computer readable media of claim 3, wherein said temporal features are further defined by at least one feature description selected from the group consisting of start time, end time and duration.

8. The computer readable media of claim 1, wherein said object hierarchy descriptions are based on visual feature relationships of video objects represented by said video object descriptions.

9. The computer readable media of claim 1, wherein said video object hierarchy descriptions are based on semantic feature relationships of video objects represented by said video object descriptions.

10. The computer readable media of claim 1, wherein said video object hierarchy descriptions are based on media feature relationships of video objects represented by said video object descriptions.

11. The computer readable media of claim 1, wherein said video object hierarchy descriptions are based on temporal feature relationships of video objects represented by said video object descriptions.

12. The computer readable media of claim 1, wherein said video object hierarchy descriptions are based on relationships of video objects represented by said video object descriptions, wherein said video object hierarchy descriptions have a plurality of hierarchal levels.

13. The computer readable media of claim 12, wherein said video object hierarchy descriptions having a plurality of hierarchal levels comprise clustering hierarchies.

14. The computer readable media of claim 13, wherein said clustering hierarchies are based on relationships of video objects represented by said video object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships, temporal feature relationships and media feature relationships.

15. The computer readable media of claim 14, wherein said video object hierarchy descriptions having a plurality of hierarchical levels are configured to comprise multiple levels of abstraction hierarchies.

16. The computer readable media of claim 15, wherein said multiple levels of abstraction hierarchies are configured to be based on relationships of video objects represented by said video object descriptions, wherein said relationships are selected from a group consisting of visual feature relationships, semantic feature relationships, temporal feature relationships and media feature relationships.

17. The computer readable media of claim 1, wherein said entity relation graph descriptions are based on relationships of video objects represented by said video object descriptions, wherein said relationships are selected from the group consisting of visual feature relationships, semantic feature relationships, temporal feature relationships and media feature relationships.

18. The computer readable media of claim 1, wherein said video object descriptions are in the form of encoded description information.

19. The computer readable media of claim 1, wherein said video object descriptions, said video object hierarchy descriptions, and said entity relation graph descriptions are combined together in the form of encoded description information.

20. The computer readable media of claim 19, wherein said encoded description information is in the form of binary encoded information.

21. The computer readable media of claim 19, wherein said encoded description information is in the form of XML encoded information.

22. The computer readable media of claim 1, wherein feature descriptions include pointers to extraction and matching code to facilitate code downloading.

23. The computer readable media of claim 5, wherein feature descriptions include pointers to extraction and matching code to facilitate code downloading.