Systems and methods for interoperable multimedia content descriptions

• US 7,653,635 B1
• Filed: 11/05/1999
• Issued: 01/26/2010
• Est. Priority Date: 11/06/1998
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Benitez, Ana, Paek, Seungup, Chang, Shih-Fu
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US09/830,899
• Time in Patent Office: 3,735 Days
• Field of Search: 707/100, 707/102, 707/1, 707/2, 707/10, 707/104.1, 345/302, 725/136, 348/461, 382/199
• US Class Current: 1/1
• CPC Class Codes: G06F 16/289 Object oriented databases
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content

Abstract

Systems and methods for generating standard description records from multimedia information are provided. The system includes at least one multimedia information input interface (180) receiving multimedia information, a computer processor, and a data storage system (150), operatively coupled to said processor, for storing said at least one description record. The processor performs object extraction processing to generate multimedia object descriptions (200, 201, 205) from the multimedia information, and object hierarchy processing (410, 420) to generate multimedia object hierarchy descriptions, to generate at least one description record including the multimedia object descriptions (200, 201, 205) and multimedia object hierarchy descriptions for content embedded within the multimedia information.

Claims

1. A system for creating a description record from multimedia information, comprising:

(a) at least one multimedia information input interface receiving said multimedia information;

(b) a computer processor, coupled to said at least one multimedia information input interface, receiving said multimedia information therefrom, processing said multimedia information by performing object extraction processing to create multimedia object descriptions from said multimedia information, said object descriptions associated with distinct multimedia objects, and processing said created multimedia object descriptions by object hierarchy processing to create multimedia object hierarchy descriptions indicative of an organization of said object descriptions and said distinct associated multimedia objects, wherein at least one description record including said multimedia object descriptions and said multimedia object hierarchy descriptions is created for content embedded within said multimedia information wherein said description record allows for searching said multimedia information for said embedded content; and

(c) a data storage system, operatively coupled to said processor, for storing said at least one description record.

2. The system of claim 1, wherein said multimedia information comprises image information, said multimedia object descriptions comprise image object descriptions, and said multimedia object hierarchy descriptions comprise image object hierarchy descriptions.

3. The system of claim 2, wherein said object extraction processing comprises:

(a) image segmentation processing to segment each image in said image information into regions within said image; and

(b) feature extraction processing to create one or more feature descriptions for one or more of said regions;

whereby said created object descriptions comprise said one or more feature descriptions for one or more of said regions.

4. The system of claim 3, wherein said one or more feature descriptions are selected from the group consisting of text annotations, color, texture, shape, size, and position.

5. The system of claim 2, wherein said object hierarchy processing comprises physical object hierarchy organization to create physical object hierarchy descriptions of said image object descriptions that are based on spatial characteristics of said objects, such that said image object hierarchy descriptions comprise physical descriptions.

6. The system of claim 5, wherein said object hierarchy processing further comprises logical object hierarchy organization to create logical object hierarchy descriptions of said image object descriptions that are based on semantic characteristics of said objects, such that said image object hierarchy descriptions comprise both physical and logical descriptions.

7. The system of claim 6, wherein said object extraction processing comprises:

(a) image segmentation processing to segment each image in said image information into regions within said image; and

(b) feature extraction processing to create object descriptions for one or more of said regions;

and wherein said physical hierarchy organization and said logical hierarchy organization create hierarchy descriptions of said object descriptions for said one or more of said regions.

8. The system of claim 7, further comprising an encoder receiving said image object hierarchy descriptions and said image object descriptions, and encoding said image object hierarchy descriptions and said image object descriptions into encoded description information, wherein said data storage system is operative to store said encoded description information as said at least one description record.

9. The system of claim 1, wherein said multimedia information comprises video information, said multimedia object descriptions comprise video object descriptions including both event descriptions and object descriptions, and said multimedia hierarchy descriptions comprise video object hierarchy descriptions including both event hierarchy descriptions and object hierarchy descriptions.

10. The system of claim 9, wherein said object extraction processing comprises:

(a) temporal video segmentation processing to temporally segment said video information into one or more video events or groups of video events and create event descriptions for said video events,

(b) video object extraction processing to segment said one or more video events or groups of video events into one or more regions, and to create object descriptions for said regions; and

(c) feature extraction processing to create one or more event feature descriptions for said one or more video events or groups of video events, and one or more object feature descriptions for said one or more regions;

wherein said created video object descriptions include said event feature descriptions and said object descriptions.

11. The system of claim 10, wherein said one or more event feature descriptions are selected from the group consisting of text annotations, shot transition, camera motion, time and key frame, and wherein said one or more object feature descriptions are selected from the group consisting of color, texture, shape, size, position, motion, and time.

12. The system of claim 9, wherein said object hierarchy processing comprises physical event hierarchy organization to create physical event hierarchy descriptions of said video object descriptions that are based on temporal characteristics of said video objects, such that said video hierarchy descriptions comprise temporal descriptions.

13. The system of claim 12, wherein said object hierarchy processing further comprises logical event hierarchy organization to create logical event hierarchy descriptions of said video object descriptions that are based on semantic characteristics of said video objects, such that said hierarchy descriptions comprise both temporal and logical descriptions.

14. The system of claim 13, wherein said object hierarchy processing further comprises physical and logical object hierarchy extraction processing, receiving said temporal and logical descriptions and creating object hierarchy descriptions for video objects embedded within said video information, such that said video hierarchy descriptions comprise temporal and logical event and object descriptions.

15. The system of claim 14, wherein said object extraction processing comprises:

(a) temporal video segmentation processing to temporally segment said video information into one or more video events or groups of video events and create event descriptions for said video events,

(b) video object extraction processing to segment said one or more video events or groups of video events into one or more regions, and to create object descriptions for said regions; and

(c) feature extraction processing to create one or more event feature descriptions for said one or more video events or groups of video events, and one or more object feature descriptions for said one or more regions;

wherein said created video object descriptions include said event feature descriptions and said object descriptions, and wherein said physical event hierarchy organization and said logical event hierarchy organization create hierarchy descriptions from said event feature descriptions, and wherein said physical object hierarchy organization and said logical object hierarchy organization create hierarchy descriptions from said object feature descriptions.

16. The system of claim 15, further comprising an encoder receiving said video object hierarchy descriptions and said video object descriptions, and encoding said video object hierarchy descriptions and said video object descriptions into encoded description information, wherein said data storage system is operative to store said encoded description information as said at least one description record.

17. A method for creating a description record from multimedia information, comprising the steps of:

(a) receiving said multimedia information;

(b) processing by a computer processor said multimedia information by performing object extraction processing to create multimedia object descriptions from said multimedia information said object descriptions associated with distinct multimedia objects;

(c) processing said created multimedia object descriptions by object hierarchy processing to create multimedia object hierarchy descriptions indicative of an organization of said object descriptions and said distinct associated multimedia objects, wherein at least one description record including said multimedia object descriptions and said multimedia object hierarchy descriptions is created for content embedded within said multimedia information wherein said description record allows for searching said multimedia information for said embedded content; and

(d) storing said at least one description record.

18. The method of claim 17, wherein said multimedia information comprises image information, said multimedia object descriptions comprise image object descriptions, and said multimedia object hierarchy descriptions comprise image object hierarchy descriptions.

19. The method of claim 18, wherein said object extraction processing step comprises the sub-steps of:

(a) image segmentation processing to segment each image in said image information into regions within said image; and

(b) feature extraction processing to create one or more feature descriptions for one or more of said regions;

whereby said created image object descriptions comprise said one or more feature descriptions for one or more of said regions.

20. The method of claim 19, wherein said one or more feature descriptions are selected from the group consisting of text annotations, color, texture, shape, size, and position.

21. The method of claim 18, wherein said step of object hierarchy processing includes the sub-step of physical object hierarchy organization to create physical object hierarchy descriptions of said image object descriptions that are based on spatial characteristics of said objects, such that said image hierarchy descriptions comprise physical descriptions.

22. The method of claim 21, said step of object hierarchy processing further includes the sub-step of logical object hierarchy organization to create logical object hierarchy descriptions of said image object descriptions that are based on semantic characteristics of said objects, such that said image object hierarchy descriptions comprise both physical and logical descriptions.

23. The method of claim 22, wherein said step of object extraction processing further includes the sub-steps of:

(a) image segmentation processing to segment each image in said image information into regions within said image; and

(b) feature extraction processing to create object descriptions for one or more of said region;

and wherein said physical object hierarchy organization sub-step and said logical object hierarchy organization sub-step create hierarchy descriptions of said object descriptions for said one or more of said regions.

24. The method of claim 18, further comprising the step of encoding said image object descriptions and said image object hierarchy descriptions into encoded description information prior to said data storage step.

25. The method of claim 17, wherein said multimedia information comprises video information, said multimedia object descriptions comprise video object descriptions including both event descriptions and object descriptions, and said multimedia hierarchy descriptions comprise video object hierarchy descriptions including both event hierarchy descriptions and object hierarchy descriptions.

26. The method of claim 25, wherein said step of object extraction processing comprises the sub-steps of:

(a) temporal video segmentation processing to temporally segment said video information into one or more video events or groups of video events and create event descriptions for said video events,

(b) video object extraction processing to segment said one or more video events or groups of video events into one or more regions, and to create object descriptions for said regions; and

(c) feature extraction processing to create one or more event feature descriptions for said one or more video events or groups of video events, and one or more object feature descriptions for said one or more regions;

wherein said created video object descriptions include said event feature descriptions and said object descriptions.

27. The method of claim 26, wherein said one or more event feature descriptions are selected from the group consisting of text annotations, shot transition, camera motion, time and key frame, and wherein said one or more object feature descriptions are selected from the group consisting of color, texture, shape, size, position, motion, and time.

28. The method of claim 25, wherein said step of object hierarchy processing includes the sub-step of physical event hierarchy organization to create physical event hierarchy descriptions of said video object descriptions that are based on temporal characteristics of said video objects, such that said video hierarchy descriptions comprise temporal descriptions.

29. The method of claim 28, wherein said step of object hierarchy processing further includes the sub-step of logical event hierarchy organization to create logical event hierarchy descriptions of said video object descriptions that are based on semantic characteristics of said video objects, such that said hierarchy descriptions comprise both temporal and logical descriptions.

30. The method of claim 29, wherein said step of object hierarchy processing further comprises the sub-step physical and logical object hierarchy extraction processing, receiving said temporal and logical descriptions and creating object hierarchy descriptions for video objects embedded within said video information, such that said video hierarchy descriptions comprise temporal and logical event and object descriptions.

31. The method of claim 30, wherein said step of object extraction processing comprises the sub-steps of:

(a) temporal video segmentation processing to temporally segment said video information into one or more video events or groups of video events and create event descriptions for said video events,

(b) video object extraction processing to segment said one or more video events or groups of video events into one or more regions, and to create object descriptions for said regions; and

(c) feature extraction processing to create one or more event feature descriptions for said one or more video events or groups of video events, and one or more object feature descriptions for said one or more regions;

wherein said created video object descriptions include said event feature descriptions and said object descriptions, and wherein said physical event hierarchy organization and said logical event hierarchy organization create hierarchy descriptions from said event feature descriptions, and wherein said physical object hierarchy organization and said logical object hierarchy organization create hierarchy descriptions from said object feature descriptions.

32. The method of claim 31, further comprising the step of encoding said video object descriptions and said video object hierarchy descriptions into encoded description information prior to said data storage step.