Method and apparatus for efficiently representing storing and accessing video information

• US 6,956,573 B1
• Filed: 11/14/1997
• Issued: 10/18/2005
• Est. Priority Date: 11/15/1996
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Kumar, Rakesh, Sawhney, Harpreet S., Carlson, Curtis R., Bergen, James Russell
• Primary Examiner(s): Jankus, Almis R.
• Application Number:
  US08/970,889
• Time in Patent Office: 2,895 Days
• Field of Search: 345/327, 345/473, 382/284, 382/220, 382/305, 382/236, 715/716
• US Class Current: 345/473
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/440227 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/8153 comprising still images, e.g. texture, background image
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal

Abstract

A method and concomitant apparatus for comprehensively representing video information in a manner facilitating indexing of the video information. Specifically, a method according to the inveniton comprises the steps of dividing a continuous video stream into a plurality of video scenes; and at least one of the steps of dividing, using intra-scene motion analysis, at least one of the plurality of scenes into one or more layers; representing, as a mosaic, at least one of the pluraliy of scenes; computing, for at least one layer or scene, one or more content-related appearance attributes; and storing, in a database, the content-related appearance attributes or said mosaic representations.

Claims

1. A method for comprehensively representing video information in a manner facilitating indexing of the video information, comprising the step of:

segmenting a video stream into a plurality of scenes, each of said scenes comprising at least one video frame;

dividing, using intra-scene motion analysis, at least one of said plurality of scenes into at least one scene foreground layer and a scene background layer;

representing each scene background layer as a mosaic, said background layer mosaic defining a key frame of a respective scene; and

representing each of said at least one video frames forming said scene as a difference between initial video frame imagery and a respective portion of said key frame.

2. The method of claim 1, further comprising the steps of:

computing, for at least one of said scene foreground and background layers, one or more content-related appearance attributes; and

storing, in a database, said scene content-related appearance attributes or said mosaic representations.

3. The method of claim 2, further comprising the steps of

storing representations of said plurality of scenes in a mass storage unit;

and

retrieving, in response to a database query, scenes associated with content-related appearance attributes defined in said database query.

4. The method of claim 1, wherein said mosaic representation comprises one of a two dimensional mosaic, a three dimensional mosaic and a network of mosaics.

5. The method of claim 2, wherein said step of computing a content-based appearance attribute for a layer of a scene comprises the steps of:

generating an image pyramid of said layer;

filtering, using one or more filters associated with said content-based appearance attribute, each subband of said image pyramid to produce respective one or more feature maps associated with each subband; and

integrating said one or more feature maps associated with each respective subband to produce respective attribute pyramid subbands, wherein each of said attribute pyramid, subbands comprises a content-based appearance attribute subband associated with a corresponding image pyramid subband.

6. The method of claim 5, wherein said content-based appearance attribute comprises at least one of a luminance attribute, a chrominance attribute and a texture attribute.

7. The method of claim 5, wherein said step of filtering further comprises the step of:

rectifying each of said one or more feature maps associated with each subband.

8. The method of claim 5, further comprising the step of:

collapsing said attribute pyramid subbands to produce a content-based appearance attribute.

9. The method of claim 2, further comprising the step of:

receiving a request for video information substantially matching a desired content-related appearance attribute; and

retrieving video frames or scenes having at least one layer associated with content-related appearance attributes substantially matching said desired content-related appearance attribute.

10. The method of claim 9, wherein said step of receiving a request comprises the steps of

identifying a query type and a query specification, said query type comprising one of a luminance, chrominance and texture query type, said query specification defining a desired property of said identified query type;

selecting a predetermined filter type associated with said identified query type; and

calculating, using said predetermined filter type and said desired property, a desired content-related appearance attribute, said desired content-related appearance attribute being suitable for comparing to said content-related appearance attributes stored in said database.

11. The method of claim 1, further comprising the steps of:

storing, in a database, ancillary information associated with one or more layers or frames of one or more scenes.

12. The method of claim 1, wherein said step of segmenting comprises the steps of:

generating a descriptor vector of a predetermined type for each video frame of a video information stream;

calculating a difference between descriptor vectors of successive frames; and

generating a scene cut indicium in response to said calculated difference exceeding a threshold level.

13. The method of claim 1, wherein said step of segmenting comprises the steps of:

generating, in a first pass, a descriptor vector of a predetermined type for each video frame of a video information stream;

calculating, using said generated descriptor vectors, a descriptor vector threshold level;

calculating, in a second pass, a difference between descriptor vectors of successive frames; and

generating a scene cut indicium in response to said calculated difference exceeding a threshold level.

14. A method for browsing a video program stored in a mass storage unit, said video program comprising a plurality of scenes, said method comprising the steps of:

providing a database associated with the stored video program, said database comprising attribute information associated with at least a representative portion of said plurality of video frames forming each scene, where said plurality of scenes comprise a plurality of video frames including a key frame comprising a mosaic of an intra-scene background layer;

formulating a query utilizing attribute information associated with a desired video frame;

searching said database to identify video frames substantially satisfying said query; and

retrieving, from said mass storage unit, one or more of said identified video frames.

15. The method of claim 14, wherein said step of formulating a query comprises the steps of:

selecting a query type;

selecting a query specification; and

computing a multi-dimensional feature vector using said query type and query specification.

16. The method of claim 15, wherein said query specification is selected by identifying a portion of a displayed image, and said multi-dimensional feature vector is calculated using said query type and said identified potion of said displayed image.

17. The method of claim 16, further comprising the steps of:

formatting, for subsequent presentation on a display device, each scene Including one or more of said identified video frames; and

transmitting said formatted scenes.

18. A computer-readable medium having stored thereon a plurality of

instructions, the plurality of instructions including instructions which,

when executed by a processor, cause the processor to perform the steps of:

(a) segmenting a video stream into a plurality of scenes, each of said scenes comprising at least one video frame;

(b) dividing, using intra-scene motion analysis, at least one of said plurality of scenes into at least one scene foreground layer and a scene background layer;

representing each scene background layer as a mosaic, said background layer mosaic defining a key frame of a respective scene; and

representing each of said at least one video frames forming said scene as a difference between initial video frame imagery and a respective portion of said key frame.

19. The computer-readable medium of claim 18, further having stored thereon an additional plurality of instructions, the additional plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the additional steps of:

computing, for at least one of said scene foreground and background layers, one or more content-related appearance attributes; and

storing, in a database, said scene content-related appearance attributes or said mosaic representations.

20. The computer-readable medium of claim 19, further having stored thereon an additional plurality of instructions, the additional plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the additional steps of:

storing representations of said plurality of scenes in a mass storage unit; and

retrieving, in response to a database query, scenes associated with content-related appearance attributes defined in said database query.

21. The computer-readable medium of claim 19, wherein said mosaic representation comprises one of a two dimensional mosaic, a three dimensional mosaic and a network of mosaics.

22. The computer-readable medium of claim 19, wherein the stored instruction of computing a content-based appearance attribute for a layer of a scene, when executed by a processor, cause the processor to perform the steps of:

generating an image pyramid of said layer,

filtering, using one or more filters associated with said content-based appearance attribute, each subband of said image pyramid to produce respective one or more feature maps associated with each subband; and

integrating said one or more feature maps associated with each respective subband to produce respective attribute pyramid subbands, wherein each of said attribute pyramid subbands comprises a content-based appearance attribute subband associated with a corresponding image pyramid subband.

23. The computer-readable medium of claim 22, wherein said content-based appearance attribute comprises at least one of a luminance attribute, a chrominance attribute and a texture attribute.