Methods and apparatuses for interactive similarity searching, retrieval, and browsing of video

• US 6,774,917 B1
• Filed: 03/11/1999
• Issued: 08/10/2004
• Est. Priority Date: 03/11/1999
• Status: Expired due to Term

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.), Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.), Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Girgensohn, Andreas, Wilcox, Lynn, Foote, Jonathan T.
• Primary Examiner(s): Nguyen, Cao (Kevin)
• Application Number:
  US09/266,558
• Time in Patent Office: 1,979 Days
• Field of Search: 345/700, 345/716, 345/723, 345/762, 345/765, 345/744, 345/835, 345/848, 345/853
• US Class Current: 715/700
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7864 using domain-transform features, e.g. DCT or wavelet transform coefficients
• CPC Class Codes: G06V 20/48 Matching video sequences

Abstract

Method for interactive selecting video consisting of training images from a video for a video similarity search and for displaying the results of the similarity search are disclosed. The user selects a time interval in the video as a query definition of training images for training an image class statistical model. Time intervals can be as short as one frame or consist of disjoint segments or shots. A statistical model of the image class defined by the training images is calculated on-the-fly from feature vectors extracted from transforms of the training images. For each frame in the video, a feature vector is extracted from the transform of the frame, and a similarity measure is calculated using the feature vector and the image class statistical model. The similarity measure is derived from the likelihood of a Gaussian model producing the frame. The similarity is then presented graphically, which allows the time structure of the video to be visualized and browsed. Similarity can be rapidly calculated for other video files as well, which enables content-based retrieval by example. A content-aware video browser featuring interactive similarity measurement is presented. A method for selecting training segments involves mouse click-and-drag operations over a time bar representing the duration of the video; similarity results are displayed as shades in the time bar. Another method involves selecting periodic frames of the video as endpoints for the training segment.

Claims

1. A method of performing a similarity search of a video, the method comprising the steps of:

interactively defining a training video segment from the video;

obtaining reduced feature vectors corresponding to frames of the training video segment;

training a statistical model using the reduced feature vectors; and

displaying a similarity measure of each frame in the video to the training video segment;

where interactively defining the training segment comprises;

providing a display window for viewing the video;

displaying a time bar within a video browser, wherein position within the time bar linearly corresponds to elapsed time from a beginning of the video; and

receiving user training input indicating one or more training video segments from the video.

2. A method as in claim 1, further comprising the steps of:

for each frame of the video,

obtaining a reduced feature vector; and

computing a similarity score using the reduced feature vector and the statistical model.

3. A method as in claim 2, further comprising the step of:

segmenting the video into similar and non-similar segments based upon the similarity scores.

4. A method as in claim 3,

wherein the step of segmenting the video into similar and non-similar segments based upon the similarity scores is performed by comparing the similarity scores to an interactively defined similarity threshold.

5. A method as in claim 2,

wherein the steps of obtaining reduced feature vectors corresponding to frames of the training video segment and, for each frame of the video, obtaining a reduced feature vector are performed by retrieval of the reduced feature vectors from a precomputed feature vector database corresponding to the video.

6. A method as in claim 2,

wherein the steps of obtaining reduced feature vectors corresponding to frames of the training video segment and, for each frame of the video, obtaining a reduced feature vector are performed transforming frames of the video.

7. A method as in claim 1,

wherein each reduced feature vector corresponding to a frame of the training video segment includes features representing chromatic components of the frame and features representing luminance components of the frame.

8. A method as in claim 7,

wherein each reduced feature vector includes fewer features representing chromatic components than features representing luminance components.

9. A method as in claim 1,

wherein each reduced feature vector corresponding to a frame of the training video segment includes features representing red components of the frame, features representing green components of the frame, and features representing blue components of the frame.

10. A method of presenting a video within a video browser, comprising the steps of:

providing a display window for viewing the video;

displaying a time bar within the video browser, wherein position within the time bar linearly corresponds to elapsed time from a beginning of the video;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each frame in the video to the training video segment using shades of the time bar at positions corresponding to each frame to indicate the similarity measure.

11. A method as in claim 10,

wherein the step of receiving user training input comprises:

receiving user training mouse input along the time bar.

12. A method as in claim 10, further comprising the step of:

receiving user threshold input indicating a threshold level for comparison with the similarity measure of each frame for labeling each frame as similar or non-similar to the one or more training video segments;

wherein a first contrasting color or pattern in the time bar indicates one of similar or non-similar and a second contrasting color or pattern indicates another of similar or non-similar.

13. A method as in claim 12, further comprising the step of:

indexing each similar frame which follows a non-similar frame as a beginning of a similar segment.

14. A method as in claim 12,

wherein the step of receiving user threshold input comprises:

receiving user threshold mouse input along a threshold slider bar.

15. A method as in claim 12, wherein the first contrasting color or pattern is a black shade, and the second contrasting color or pattern is a white shade.

16. A method of presenting a video within a web-based interface, comprising the steps of:

displaying periodic frames of the video separated by a predetermined time interval;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each displayed periodic frame in the video to the training video segment using shades surrounding each displayed periodic frame to indicate the similarity measure.

17. A method as in claim 16,

wherein the step of receiving user training input comprises:

receiving user training mouse input by detecting mouse clicks on adjacently displayed periodic frames.

18. A computer readable storage medium, comprising:

computer readable program code embodied on said computer readable storage medium, said computer readable program code for programming a computer to perform a method of performing a similarity search of a video, the method comprising the steps of;

interactively defining a training video segment from the video;

obtaining reduced feature vectors corresponding to frames of the training video segment;

training a statistical model using the reduced feature vectors; and

displaying a similarity measure of each frame in the video to the training video segment;

wherein interactively defining the training segment comprises;

providing a display window for viewing the video;

displaying a time bar within a video browser, wherein position within the time bar linearly corresponds to elapsed time from a beginning of the video; and

receiving user training input indicating one or more training video segments from the video.

19. A computer readable storage medium, comprising:

computer readable program code embodied on said computer readable storage medium, said computer readable program code for programming a computer to perform a method of presenting a video within a video browser, comprising the steps of;

providing a display window for viewing the video;

displaying a time bar within the video browser, wherein position within the time bar linearly corresponds to elapsed time from a beginning of the video;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each frame in the video to the training video segment using shades of the time bar at positions corresponding to each frame to indicate the similarity measure.

20. A computer readable storage medium, comprising:

computer readable program code embodied on said computer readable storage medium, said computer readable program code for programming a computer to perform a method of presenting a video within a web-based interface, comprising the steps of;

displaying periodic frames of the video separated by a predetermined time interval;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each displayed periodic frame in the video to the training video segment using shades surrounding each displayed periodic frame to indicate the similarity measure.

21. A computer system, comprising:

a processor;

a user interface; and

a processor readable storage medium having processor readable program code embodied on said processor readable storage medium, said processor readable program code for programming the computer system to perform a method of performing a similarity search of a video, the method comprising the steps of;

interactively defining a training video segment from the video;

obtaining reduced feature vectors corresponding to frames of the training video segment; and

training a statistical model using the reduced feature vectors.

22. A computer system, comprising:

a display;

a user interface;

a processor; and

a processor readable storage medium having processor readable program code embodied on said processor readable storage medium, said processor readable program code for programming the computer system to perform a method of presenting a video within a video browser, comprising the steps of;

providing a display window for viewing the video;

displaying a time bar within the video browser, wherein position within the time bar linearly corresponds to elapsed time from a beginning of the video;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each frame in the video to the training video segment using shades of the time bar at positions corresponding to each frame to indicate the similarity measure.

23. A computer system, comprising:

a display;

a user interface;

a processor; and

a processor readable storage medium having processor readable program code embodied on said processor readable storage medium, said processor readable program code for programing the computer system to perform a method of presenting a video within a web-based interface, comprising the steps of;

displaying periodic frames of the video separated by a predetermined time interval;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each displayed periodic frame in the video to the training video segment using shades surrounding each displayed periodic frame to indicate the similarity measure.