Interactive montages of sprites for indexing and summarizing video

• US 7,982,738 B2
• Filed: 12/01/2004
• Issued: 07/19/2011
• Est. Priority Date: 12/01/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Pal, Chris, Jojic, Nebojsa
• Primary Examiner(s): Broome; Said
• Application Number:
  US11/004,760
• Publication Number:
  US 20060117356A1
• Time in Patent Office: 2,421 Days
• Field of Search: None
• US Class Current: 345/473
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44029 for generating different versions
• CPC Class Codes: H04N 21/45452 applied to an object-based stream, e.g. MPEG-4 streams
• CPC Class Codes: H04N 21/8193 dedicated tools, e.g. video decoder software or IPMP tool
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: Y10S 345/95 Sprite processing

Abstract

A “Video Browser” provides interactive browsing of unique events occurring within an overall video recording. In particular, the Video Browser processes the video to generate a set of video sprites representing unique events occurring within the overall period of the video. These unique events include, for example, motion events, security events, or other predefined event types, occurring within all or part of the total period covered by the video. Once the video has been processed to identify the sprites, the sprites are then arranged over a background image extracted from the video to create an interactive static video montage. The interactive video montage illustrates all events occurring within the video in a single static frame. User selection of sprites within the montage causes either playback of a portion of the video in which the selected sprites were identified, or concurrent playback of the selected sprites within a dynamic video montage.

Claims

1. A system for viewing and interacting with video sprites generated from a video stream, comprising:

a device for receiving a video stream,

a device for processing the video stream to generate a set of one or more video sprites, with each video sprite corresponding to a separate unique event detected in the video stream;

a device for extracting a common background image from the video stream;

a device for generating at least one static sprite montage of the video sprites by arranging one or more of the video sprites in fixed positions on a copy of the common background image to create each static sprite montage;

a device for generating at least one dynamic sprite montage by creating an animation of one or more of the video sprites in each static sprite montage against a copy of the common background image;

a device for displaying one or more of the static sprite montages in an interactive user interface; and

a device for displaying one or more of dynamic sprite montages in the interactive user interface.

2. The system of claim 1 wherein the interactive user interface further includes a video playback window for playing at least a portion of the video stream.

3. The system of claim 2 wherein each of the sprites within the at least one static sprite montage are user selectable via the user interface, and wherein user selection of any of the sprites initiates a playback of a segment of the video stream in the video playback window corresponding to image frames of the video stream from which the selected sprite was generated.

4. The system of claim 2 wherein each of the sprites within the at least one dynamic sprite montage are user selectable via the user interface, and wherein user selection of any of the sprites initiates a playback of a segment of the video stream in the video playback window corresponding to image frames of the video stream from which the selected sprite was generated.

5. The system of claim 1 wherein generating at least one dynamic sprite montage further comprises automatically populating the at least one dynamic sprite montage with video sprites corresponding to video sprites selected from the at least one static sprite montage via the user interface.

6. The system of claim 1 wherein generating at least one dynamic sprite montage further comprises automatically populating the at least one dynamic sprite montage with video sprites corresponding to video sprites generated from a predetermined segment of the video stream.

7. The system of claim 1 wherein generating at least one dynamic sprite montage further comprises automatically populating the at least one dynamic sprite montage with all of the video sprites used to generate the at least one static sprite montage.

8. The system of claim 1 further comprising including a visible time stamp in close proximity to each of the sprites within the at least one static sprite montage, and wherein the visible time stamp corresponds to a time that the corresponding unique event was detected in the video stream.

9. The system of claim 1 further comprising including a visible time stamp in close proximity to each of the sprites within the at least one dynamic sprite montage, and wherein the visible time stamp corresponds to a time that the corresponding unique event was detected in the video stream.

10. The system of claim 1 wherein the interactive user interface provide a control for processing user queries relating to the video sprites generated from the video stream.

11. The system of claim 10 wherein the user queries include sprite matching queries.

12. The system of claim 10 wherein the user queries include sprite positional queries with respect to a location within one or more video frames of the video stream wherein corresponding unique events were detected in the video stream.

13. The system of claim 10 wherein the user queries include a sprite observation time window, wherein sprite observation times correspond to a time wherein corresponding unique events were detected in the video stream.

14. The system of claim 1 wherein extracting the common background image from the video stream comprises automatically constructing a panoramic image from one or more frames of the video stream.

15. The system of claim 1 wherein generating at least one static sprite montage further includes populating one or more of the static sprite montages with one or more user selectable sprite thumbnails.

16. The system of claim 1 wherein arranging one or more of the video sprites in fixed positions on a copy of the common background image to create each static sprite montage comprises adjusting a position of one or more of the sprites along a computed trajectory of each sprite so as to minimize potential sprite overlap within the static sprite montage.

17. The system of claim 1 wherein creating an animation of one or more of the video sprites on a copy of the common background image for generating the at least one dynamic sprite montage further comprises staggering an individual start time for the animation of each individual sprite in the animation so as to minimize potential sprite overlap within the dynamic sprite montage during animation of the dynamic sprite montage.

18. The system of claim 1 wherein generating at least one static sprite montage further includes allowing sets of one or more of the sprites to at least partially overlap in accordance with an estimated z-buffer index computed for each video sprite.

19. The system of claim 1 wherein creating an animation of one or more of the video sprites a copy of the common background for generating the at least one dynamic sprite montage further includes allowing sets of one or more of the sprites to at least partially overlap at one or more times during animation of the dynamic sprite montage in accordance with an estimated z-buffer index computed for each video sprite.

20. A computing device having a user interface for interacting with a prerecorded video stream, comprising:

providing at least one window within a user interface containing an automatically generated static representation of user selectable video sprites generated from a video stream against a background image extracted from the video stream;

providing at least one window within the user interface containing an automatically generated animated representation of one or more of the user selectable video sprites against the background image; and

providing at least one video playback window within the user interface for presenting segments of the video stream corresponding to user selected video sprites.

21. The computing device of claim 20 wherein generating video sprites from the video stream comprises automatically generating at least one unique video sprite corresponding to each of a plurality of automatically detected motion events observed within the video stream.

22. The computing device of claim 20 wherein automatically generating the static representation of user selectable video sprites comprises arranging one or more of the video sprites in fixed positions on the background image, and wherein possible overlap between two or more of the video sprites is minimized by shifting one or more of the video sprites along a computed trajectory line corresponding to each video sprite.

23. The computing device of claim 22 wherein the possible overlap between video sprites is further minimized by scaling one or more of the video sprites to further reduce video sprite overlap.

24. The computing device of claim 22 wherein any overlap between video sprites is managed by overlapping the video sprites in accordance with an estimated z-buffer index computed for each video sprite.

25. The computing device of claim 20 wherein a visible time stamp is included in close proximity to each of the video sprites within any of the at least one static representation of user selectable video sprites and the at least one animated representation of one or more of the user selectable video sprites.

26. The computing device of claim 20 wherein the user interface further includes a query control for processing user queries relating to the video sprites generated from the video stream, and wherein said queries include any of sprite matching queries, sprite positional queries, and sprite observation time queries.

27. The computing device of claim 20 wherein automatically generating the animated representation of one or more of the user selectable video sprites further comprises staggering an individual start time for the animation of each individual sprite in the animation so as to minimize potential sprite overlap within the animation.

28. The computing device of claim 27 wherein sprite overlap during animation of the dynamic sprite montage is managed in accordance with an estimated z-buffer index computed for each video sprite.

29. A computer-readable storage medium not including a modulated data signal having computer executable instructions for providing a user interface for interacting with a video stream, said computer executable instructions comprising:

recording a video of a scene using at least one camera,

processing the video to detect one or more unique events occurring within the video, and designating video frames corresponding to each detected event as an event data set;

processing each event data set to generate a set of one or more video sprites, with each video sprite corresponding to unique events represented by each corresponding event data set;

extracting a common background image from the video;

generating at least one static sprite montage of the video sprites by arranging one or more of the video sprites in fixed positions on a copy of the common background image to create each static sprite montage, and wherein each of the video sprites is user selectable;

generating at least one dynamic sprite montage by creating an animation of one or more of the video sprites from one or more of the static sprite montages against a copy of the common background image, and wherein each of the animated video sprites is user selectable;

displaying one or more of the static sprite montages in an interactive user interface;

displaying one or more of dynamic sprite montages in the interactive user interface; and

providing a video playback window in the interactive user interface for providing a playback of at least a portion of the video stream corresponding to video frames associated with the event data set corresponding to any user selected video sprites.

30. The computer-readable storage medium of claim 29 wherein generating at least one dynamic sprite montage further comprises automatically populating the at least one dynamic sprite montage with video sprites corresponding to video sprites selected from the at least one static sprite montage via the user interface.

31. The computer-readable storage medium of claim 29 further comprising including a visible time stamp in close proximity to each of the video sprites within any of the at least one static sprite montage and the at least one dynamic sprite montage.

32. The computer-readable storage medium of claim 29 further comprising including a visible time stamp in close proximity to each of the sprites within the at least one dynamic sprite montage, and wherein the visible time stamp corresponds to a time that the corresponding unique event was detected in the video stream.

33. The computer-readable storage medium of claim 29 wherein the user interface further includes a query control for processing user queries relating to the video sprites, and wherein the user queries include any of sprite matching queries, sprite positional queries, and sprite observation time queries.

34. The computer-readable storage medium of claim 29 wherein generating at least one static sprite montage further includes populating one or more of the static sprite montages with one or more user selectable sprite thumbnails.

35. The computer-readable storage medium of claim 29 wherein arranging one or more of the video sprites in fixed positions on a copy of the common background image to create each static sprite montage comprises adjusting a position of one or more of the sprites along a computed trajectory of each sprite so as to minimize potential sprite overlap within the static sprite montage.

36. The computer-readable storage medium of claim 29 wherein creating an animation of one or more of the video sprites on a copy of the common background for generating the at least one dynamic sprite montage further comprises staggering an individual start time for the animation of each individual sprite in the animation so as to minimize potential sprite overlap within the dynamic sprite montage during animation of the dynamic sprite montage.