System and method for video browsing using a cluster index

• US 20060120624A1
• Filed: 12/08/2004
• Published: 06/08/2006
• Est. Priority Date: 12/08/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Jojic, Nebojsa, Basu, Sumit
• Granted Patent: US 7,594,177 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence

Abstract

A “Video Browser” provides an intuitive user interface for indexing, and interactive visual browsing, of particular elements within a video recording. In general, the Video Browser operates by first generating a set of one or more mosaic images from the video recording. In one embodiment, these mosaics are further clustered using an adjustable similarity threshold. User selection of a particular video mosaic then initiates a playback of corresponding video frames. However, in contrast to conventional mosaicing schemes which simply play back the set of frames used to construct the mosaic, the Video Browser provides a playback of only those individual frames within which a particular point selected within the image mosaic was observed. Consequently, user selection of a point in one of the image mosaics serves to provide a targeted playback of only those frames of interest, rather than playing back the entire image sequence used to generate the mosaic.

Claims

1. A computer-readable medium having computer executable instructions for providing an interactive graphical user interface for browsing a video recording, said computer executable instructions comprising:

analyzing a video recording to identify one or more clusters of similar image scenes, with each image scene comprising at least one image frame;

generating a panoramic image mosaic for each cluster of similar image scenes;

constructing a corresponding frame map for each panoramic image mosaic, wherein each frame map provides a reverse mapping from each pixel of the panoramic image mosaic to every corresponding image frame of the video recording from which the pixel of the panoramic image mosaic was generated;

displaying one or more of the panoramic image mosaics within a first display window of a graphical user interface, and overlaying the corresponding frame maps on each of the displayed panoramic image mosaics;

tracking user selection of pixels of the displayed panoramic image mosaics; and

initiating a playback in a second display window of the graphical user interface of image frames corresponding to the user selected pixels with respect to the corresponding overlaid frame map.

2. The computer-readable medium of claim 1 wherein displaying one or more of the panoramic image mosaics within the first display window comprises displaying a panel comprised of a plurality of image mosaics which jointly represent at least a portion of the entirety of the video recording.

3. The computer-readable medium of claim 1 wherein identifying one or more clusters of similar image scenes comprises determining a hierarchical similarity of the image scenes such that the clusters are identified at each of a plurality of hierarchical similarity levels.

4. The computer-readable medium of claim 3 further comprising a user interface control for expanding one or more of the panoramic image mosaics to display a plurality of related panoramic image mosaics representing clusters of similar image scenes at a lower level of the hierarchical similarity levels.

5. The computer-readable medium of claim 3 further comprising a user interface control for collapsing two or more of the panoramic image mosaics into a single panoramic image mosaic representing a cluster of similar image scenes at a higher level of the hierarchical similarity levels.

6. The computer-readable medium of claim 3 further comprising a user adjustable similarity threshold for automatically adjusting a current hierarchical similarity level of clusters of similar image scenes, thereby automatically adjusting the display of corresponding panoramic image mosaics within the first display window.

7. The computer-readable medium of claim 1 further comprising a user interface control for zooming a selected part of one or more panoramic image mosaics in the first display window of the graphical user interface.

8. The computer-readable medium of claim 1 wherein displaying one or more of the panoramic image mosaics within a first display window of the graphical user interface comprises arranging each of the panoramic image mosaics in a scrollable display window.

9. The computer-readable medium of claim 1 further comprising a timeline display window within the graphical user interface for simultaneously highlighting a relative temporal position of each of the image frames corresponding to the user selected pixels with respect to the corresponding overlaid frame map.

10. The computer-readable medium of claim 8 further comprising additionally highlighting the relative temporal position of the currently playing image frame relative to the other highlighted relative positions of the image frames.

11. The computer-readable medium of claim 1 further comprising a video position slider bar for indicating a current playback position of the video recording.

12. The computer-readable medium of claim 11 wherein the video position slider bar provides for real-time adjustment of the current playback position of the video recording.

13. The computer-readable medium of claim 1 further comprising a control for creating and displaying a subset of panoramic mosaic images and corresponding frame maps from one or more video recordings to create an interactive video scrapbook.

14. The computer-readable medium of claim 1 further comprising automatically muting a playback of a sound track corresponding to the image frames corresponding to the user selected pixels while a pixel selection device is being moved relative to the pixels of the displayed panoramic image mosaics.

15. The computer-readable medium of claim 1 further comprising automatically varying a volume level of a sound track corresponding to the image frames corresponding to the user selected pixels in inverse proportion to the speed of the pixel selction device relative to the pixels of the displayed panoramic image mosaics.

16. The computer-readable medium of claim 14 further comprising automatically increasing a volume level of the sound track of the video recording whenever the pixel selection device is stationary relative to the to the pixels of the displayed panoramic image mosaics.

17. A system for providing interactive content-based browsing and playback of a video recording, comprising:

generating a set of hierarchically clustered panoramic image mosaics from similar scenes of the video recording, wherein each panoramic image mosaic includes a user selectable frame map overlay which identifies each image frame of the video recording that contributed to each pixel of each corresponding panoramic image mosaic;

displaying the panoramic image mosaics with a first window of a graphical user interface;

selecting a subset of one or more pixels of one of the displayed panoramic image mosaics; and

providing a playback of the image frames of the video recording corresponding to the user selected pixels relative to the corresponding frame map.

18. The system of claim 17 wherein selecting a subset of one or more pixels of one of the displayed panoramic image mosaics comprises continuously tracking a position of a pixel selection device relative to the pixels of the displayed panoramic image mosaics, and automatically selecting at least one pixel corresponding to a current pixel selection device position.

19. The system of claim 17 wherein generating a set of hierarchically clustered panoramic image mosaics from the video recording comprises generating panoramic image mosaics at each of a plurality of hierarchical similarity levels.

20. The system of claim 19 further comprising a user interface control for expanding one or more of the panoramic image mosaics to display a plurality of related panoramic image mosaics representing clusters of similar image scenes at a lower level of the hierarchical similarity levels.

21. The system of claim 19 further comprising a user interface control for collapsing two or more of the panoramic image mosaics into a single panoramic image mosaic representing a cluster of similar image scenes at a higher level of the hierarchical similarity levels.

22. The system of claim 19 further comprising a user adjustable similarity threshold for automatically adjusting a current hierarchical similarity level of clusters of similar image scenes, thereby automatically adjusting the display of corresponding panoramic image mosaics within the first window of the graphical user interface.

23. The system of claim 17 further comprising a user interface control for zooming a selected part of one or more panoramic image mosaics in the first window of the graphical user interface.

24. The system of claim 17 wherein the first window is scrollable to provide user browsing of the displayed panoramic image mosaics.

25. The system of claim 17 further comprising a timeline display window within the graphical user interface for simultaneously highlighting a relative temporal position of each of the image frames corresponding to the user selected pixels with respect to the corresponding overlaid frame map.

26. The system of claim 17 further comprising a user interface control for creating and displaying a subset of panoramic mosaic images from one or more video recordings to create an interactive video scrapbook for browsing selected portions of the one or more video recordings via the user selectable frame map overlays corresponding to each panoramic image mosaic.

27. A computer-implemented process for joint temporal-spatial browsing of video recordings via an interactive graphical user interface, comprising using a computing device for:

analyzing a video recording to identify one or more hierarchical clusters of similar image scenes at each of a plurality of hierarchical similarity levels, with each image scene comprising at least one image frame;

generating a panoramic image mosaic for each cluster, and wherein each panoramic image mosaic includes a user selectable pixel-based frame map overlay which provides a reverse mapping from each pixel of the panoramic image mosaic to every corresponding image frames of the video recording from which each pixel of the panoramic image mosaics were generated;

displaying one or more of the panoramic image mosaics within a first display window of a graphical user interface;

selecting one or more pixels of the panoramic image; and

providing a playback of the image frames of the video recording corresponding to the selected pixels relative to the corresponding frame map.

28. The computer-implemented process of claim 27 wherein the graphical user interface further includes a control for expanding one or more of the panoramic image mosaics to display a plurality of related panoramic image mosaics representing clusters of similar image scenes at a lower level of the hierarchical similarity levels.

29. The computer-implemented process of claim 27 wherein the graphical user interface further includes a control for collapsing two or more of the panoramic image mosaics into a single panoramic image mosaic representing a cluster of similar image scenes at a higher level of the hierarchical similarity levels.

30. The computer-implemented process of claim 27 further comprising a user adjustable similarity threshold for automatically adjusting a current hierarchical similarity level of clusters of similar image scenes, thereby automatically adjusting the display of corresponding panoramic image mosaics within the first display window.

31. The computer-implemented process of claim 27 wherein the graphical user interface further includes a control for zooming a selected part of one or more panoramic image mosaics in the first display window of the graphical user interface.

32. The computer-implemented process of claim 27 further comprising a timeline display window within the graphical user interface for simultaneously highlighting a relative temporal position of each of the image frames corresponding to the selected pixels relative to the corresponding frame map.

33. The computer-implemented process of claim 27 wherein the graphical user interface further includes a control for selecting and displaying a subset of panoramic mosaic images from one or more separate video recordings to create an interactive video scrapbook for browsing selected portions of the one or more separate video recordings via the user selectable pixel-based frame map overlay corresponding to each panoramic image mosaic.

34. The computer-implemented process of claim 27 wherein displaying one or more of the panoramic image mosaics within the first display window comprises displaying a panel comprised of a plurality of image mosaics which jointly represent at least a portion of the entirety of the video recording.

35. A method in a computer system for browsing a video recording via a user interface, comprising using a computer to perform the following steps:

a step for populating a first display window with at least one panoramic image mosaic, each panoramic image mosaic being generated from a cluster of similar frames identified in a video recording;

a step for assigning a map overlay to each panoramic image mosaic, said map overlay providing a reverse mapping from each pixel of each panoramic image mosaic to each corresponding image frame of the cluster similar frames from which each pixel of the panoramic image mosaic was generated; and

a step for initiating a playback of image frames of the video recording in a second display window, said image frames corresponding to selected pixels of one of the panoramic image mosaics relative to the corresponding map overlay.

36. The method of claim 35 wherein the selection of pixels of one of the panoramic image mosaics comprises a step for monitoring pixel selection device movements relative to displayed pixels of the panoramic image mosaics, and selecting pixels proximate to a current position of the pixel selection device.

37. The method of claim 35 further comprising a step for identifying hierarchical clusters of similar frames at each of a plurality of hierarchical frame similarity levels, said hierarchical clusters used as the clusters for generating the panoramic image mosaics at each hierarchical similarity level.

38. The method of claim 37 further comprising a step for expanding one or more of the panoramic image mosaics into a subset of panoramic image mosaics corresponding to each hierarchical frame similarity level of the corresponding hierarchical cluster.

39. The method of claim 37 further comprising a step for collapsing two or more of the panoramic image mosaics into one or more panoramic image mosaics corresponding to a selected hierarchical frame similarity level of the corresponding hierarchical clusters.

40. The method of claim 36 further comprising a step for automatically muting a sound track of the video recording while the pixel selection device is being moved relative to the displayed pixels of the panoramic image mosaics.

41. The method of claim 36 further comprising a step for automatically varying a volume level of a sound track of the video recording in inverse proportion to the speed at which the pixel selection device is being moved relative to the displayed pixels of the panoramic image mosaics.

42. The method of claim 40 further comprising a step for automatically increasing a volume level of the sound track of the video recording whenever the pixel selection device is stationary relative to the displayed pixels of the panoramic image mosaics.

43. The method of claim 36 further comprising a step for automatically filtering a sound track of the video recording as a function of the motion of the pixel selection device relative to the displayed pixels of the panoramic image mosaics.

44. The method of claim 36 further comprising a step for automatically filtering a sound track of the video recording as a function of the motion of the pixel selection device relative to the displayed pixels of the panoramic image mosaics.

45. The method of claim 36 further comprising a step for automatically changing the speed of a playback of a sound track of the video recording as a function of the motion of the pixel selection device relative to the displayed pixels of the panoramic image mosaics.

46. The method of claim 45 wherein the steo for automatically changing the speed of the playback further includes a step for applying a pitch-preserving speed change to the playback of the sound track.