User interface for adaptive video fast forward

• US 20040189691A1
• Filed: 03/28/2003
• Published: 09/30/2004
• Est. Priority Date: 03/28/2003
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: Microsoft Corporation
• Inventors: Petrovic, Nemanja, Jojic, Nebojsa
• Granted Patent: US 7,152,209 B2
• Time in Patent Office: Days
• US Class Current: 715/720
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A user interface (UI) for adaptive video fast forward provides a novel fully adaptive content-based UI for allowing user interaction with an image sequence or video relative to a user identified query sample. This query sample is drawn either from an image sequence being searched or from another image sequence entirely. The user interaction offered by the UI includes providing a user with computationally efficient searching, browsing and retrieval of one or more objects, frames or sequences of interest in video or image sequences, as well as automatic content-based variable-speed playback based on a computed similarity to the query sample. In addition, the UI also provides the capability to search for image frames or sequences that are dissimilar to the query sample, thereby allowing the user to quickly locate unusual or different activity within an image sequence.

Claims

1. A computer-readable medium having computer executable instructions for providing automatic variable-speed playback of an image sequence, said computer executable instructions comprising:

selecting an input image sequence via a graphical user interface;

providing a playback of the input image sequence within a first display window within the graphical user interface;

selecting a comparison sample via the graphical user interface;

providing a graphical representation of the comparison sample in a second display window within the graphical user interface;

providing a graphical representation of one or more image frames from the input image sequence that match the comparison sample in a third display window within the graphical user interface; and

automatically varying a playback speed of the input image sequence with respect to a probabilistic likelihood of each frame of the input image sequence relative to the comparison sample.

2. The computer-readable medium of claim 1 further comprising an image sequence index window for displaying thumbnail representations of image frames extracted at regular intervals from the input image sequence.

3. The computer-readable medium of claim 2 wherein the thumbnail representations of the input image sequence are selectable via the user interface, and wherein selection of a particular thumbnail representation of the input image sequence causes the input image sequence to be played within the first display window beginning at the image frame represented by the selected thumbnail representation of the input image sequence.

4. The computer-readable medium of claim 1 wherein a determination of whether an image frame from the input image sequence matches the comparison sample comprises:

automatically learning a generative model from the comparison sample; and

comparing the frames in the input image sequence to the generative model to determine the probabilistic likelihood of each frame under the generative model.

5. The computer-readable medium of claim 4 further comprising a control for selecting image frames having a high probabilistic likelihood under the generative model as matching the comparison sample.

6. The computer-readable medium of claim 4 further comprising a control for selecting image frames having a low probabilistic likelihood under the generative model as matching the comparison sample.

7. The computer-readable medium of claim 4 further comprising a user adjustable similarity threshold for comparing the frames in the input image sequence to the generative model.

8. The computer-readable medium of claim 1 further comprising a playback speed slider bar for real-time adjustment of the playback speed of the input image sequence.

9. The computer-readable medium of claim 8 wherein the playback speed slider bar indicates the playback speed of the input image sequence.

10. The computer-readable medium of claim 1 further comprising a video position slider bar for indicating a current playback position of the input image sequence.

11. The computer-readable medium of claim 10 wherein the video position slider bar provides for real-time adjustment of the current playback position of the input image sequence.

12. The computer-readable medium of claim 1 further comprising a control for extracting a subset of consecutive image frames from the input image sequence to form the comparison sample.

13. The computer-readable medium of claim 1 further comprising a control for loading the comparison sample from at least one previously stored comparison sample.

14. The computer-readable medium of claim 13 wherein the previously stored comparison sample includes a previously learned generative model.

15. The computer-readable medium of claim 1 wherein selecting an input image sequence via the graphical user interface comprises automatically importing the input image sequence from at least one camera.

16. The computer-readable medium of claim 1 wherein selecting an input image sequence via the graphical user interface comprises automatically loading the input image sequence from a computer readable storage medium.

17. The computer-readable medium of claim 1 wherein providing the graphical representation of the comparison sample in the second display window comprises displaying a representative image frame from the comparison sample.

18. The computer-readable medium of claim 1 wherein providing the graphical representation of the comparison sample in the second display window comprises displaying a looped playback of the image frames comprising the comparison sample.

19. The computer-readable medium of claim 1 wherein the graphical representation of one or more image frames from the input image sequence that match the comparison sample are provided as user selectable thumbnail representations.

20. The computer-readable medium of claim 19 wherein selection of any of the user selectable thumbnail representations causes playback of the one or more image frames from the input image sequence that match the comparison sample within the first display window.

21. The computer-readable medium of claim 1 wherein automatically varying the playback speed of the input image sequence comprises increasing playback speed as the probabilistic likelihood decreases and decreasing payback speed as the probabilistic likelihood increases.

22. A system for providing automatic fully adaptive content-based variable-speed playback of a video, comprising:

providing a graphical user interface;

selecting a video via the graphical user interface;

selecting a number of consecutive image frames from the video as a query sample via the graphical user interface;

automatically learning a generative model from the query sample;

displaying the video in a video window within the graphical user interface;

displaying the query sample in a query sample window within the graphical user interface;

matching portions of the video to the query sample by determining a probabilistic likelihood of each image frame of the video under the generative model;

displaying the matched portions of the video as thumbnail representations within a query match window within the graphical user interface; and

automatically varying a playback speed of the video in inverse proportion to the probabilistic likelihood of each frame.

23. The system of claim 22 wherein the playback speed of the video decreases as the likelihood increases, and wherein the payback speed increases as the likelihood decreases.

24. The system of claim 23 wherein the playback speed the video is determined by a combined likelihood of one or more adjacent image frames in the video.

25. The system of claim 22 wherein the generative model is based on a user definable number of color blobs for modeling multiple objects represented in the query sample.

26. The system of claim 22 wherein the generative model is capable of modeling occluding objects.

27. The system of claim 22 further comprising comparing the likelihood of each frame under the generative model to a similarity threshold.

28. The system of claim 27 wherein image frames that have a likelihood greater than the similarity threshold are identified as being similar to the query sample, and wherein image frames that do not have a likelihood greater than the similarity threshold are identified as being dissimilar to the query sample.

29. The system of claim 27 further comprising a control for allowing user adjustment of the similarity threshold via the graphical user interface.

30. The system of claim 22 further comprising an image sequence index window for displaying thumbnail representations of image frames extracted at regular intervals from the video.

31. The system of claim 30 wherein selection of any of the thumbnail representations of the video via the graphical user interface causes the input image sequence to be played within the video window beginning at the image frame represented by the selected thumbnail representation.

32. The system of claim 22 further comprising a playback speed slider bar for real-time adjustment of the playback speed of the video.

33. The system of claim 22 further comprising a video position slider bar for real-time adjustment of the current playback position of the video.

34. The system of claim 22 further comprising a control for loading the query sample from a set of at least one previously stored query samples.

35. The system of claim 22 further comprising a control for loading the generative model from a set of at least one previously computed generative models.

36. The system of claim 22 further comprising a control for saving the learned generative model to a computer-readable medium.

37. A computer-implemented process for using a graphical user interface for automatically identifying similar image frames in one or more image sequences, comprising:

use a graphical user interface to select at least one image sequence, each image sequence having at least one image frame;

use the graphical user interface to select a query sample consisting of at least one image frame from one of the at least one image sequences;

use the graphical user interface to input a desired number of blobs to be modeled in learning a generative model of the query sample;

automatically learn the generative model from the query sample;

compare the frames in each image sequence to the generative model to determine a likelihood of each frame under the generative model; and

provide an automatic variable speed playback of at least one of the image sequences within the graphical user interface relative to the likelihood of each frame under the generative model.

38. The computer-implemented process of claim 37 wherein the likelihood of each frame under the generative model is compared to an adjustable similarity threshold.

39. The computer-implemented process of claim 38 wherein image frames that have a likelihood greater than the similarity threshold are similar to the at least one image frame of the query sample, and wherein all other image frames are dissimilar to the at least one image frame of the query sample.

40. The computer-implemented process of claim 37 wherein the playback speed the at least one image sequence is inversely proportional to the likelihood of each image frame.

41. The computer-implemented process of claim 40 wherein the playback speed decreases as the likelihood increases, and wherein the payback speed increases as the likelihood decreases.

42. The computer-implemented process of claim 37 wherein the generative model models multiple objects in the query sample as blobs, and wherein the generative model is capable of modeling occluding objects in the query sample.

43. The computer-implemented process of claim 37 wherein automatically learning the generative model from the query sample comprises performing an variational expectation-maximization (EM) analysis of each image frame in the query sample for learning a probabilistic explanation of the query sample.

44. The computer-implemented process of claim 37 further comprising an image sequence index window for displaying thumbnail representations of image frames extracted at regular intervals from the at least one image sequence.

45. The computer-implemented process of claim 44 wherein selection of any of the thumbnail representations of the at least one image sequence via the graphical user interface causes the at least one image sequence to be played within the graphical user interface beginning at the image frame represented by the selected thumbnail representation.

46. The computer-implemented process of claim 37 further comprising a playback speed slider bar for real-time adjustment of the playback speed of the at least one image sequence.

47. The computer-implemented process of claim 37 further comprising a video position slider bar for real-time adjustment of a current playback position of the at least one image sequence.