Methods and apparatuses for interactive similarity searching, retrieval and browsing of video

• US 20040221237A1
• Filed: 06/03/2004
• Published: 11/04/2004
• Est. Priority Date: 03/11/1999
• Status: Active Grant

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.), Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.), Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Girgensohn, Andreas, Wilcox, Lynn, Foote, Jonathan T.
• Granted Patent: US 7,246,314 B2
• Time in Patent Office: Days
• US Class Current: 715/700
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/7864 using domain-transform features, e.g. DCT or wavelet transform coefficients
• CPC Class Codes: G06V 20/48 Matching video sequences

Abstract

Methods for interactive selecting video queries consisting of training images from a video for a video similarity search and for displaying the results of the similarity search are disclosed. The user selects a time interval in the video as a query definition of training images for training an image class statistical model. Time intervals can be as short as one frame or consist of disjoint segments or shots. A statistical model of the image class defined by the training images is calculated on-the-fly from feature vectors extracted from transforms of the training images. For each frame in the video, a feature vector is extracted from the transform of the frame, and a similarity measure is calculated using the feature vector and the image class statistical model. The similarity measure is derived from the likelihood of a Gaussian model producing the frame. The similarity is then presented graphically, which allows the time structure of the video to be visualized and browsed. Similarity can be rapidly calculated for other video files as well, which enables content-based retrieval by example. A content-aware video browser featuring interactive similarity measurement is presented. A method for selecting training segments involves mouse click-and-drag operations over a time bar representing the duration of the video; similarity results are displayed as shades in the time bar. Another method involves selecting periodic frames of the video as endpoints for the training segment.

Claims

1. A method of performing a similarity search of a video, the method comprising the steps of:

interactively defining a training video segment from the video;

obtaining reduced feature vectors corresponding to frames of the training video segment; and

training a statistical model using the reduced feature vectors.

2. A method as in claim 1, further comprising the steps of:

for each frame of the video,

obtaining a reduced feature vector; and

computing a similarity score using the reduced feature vector and the statistical model.

3. A method as in claim 2, further comprising the step of:

segmenting the video into similar and non-similar segments based upon the similarity scores.

4. A method as in claim 2,

wherein the steps of obtaining reduced feature vectors corresponding to frames of the training video segment and, for each frame of the video, obtaining a reduced feature vector are performed by retrieval of the reduced feature vectors from a precomputed feature vector database corresponding to the video.

5. A method as in claim 2,

wherein the steps of obtaining reduced feature vectors corresponding to frames of the training video segment and, for each frame of the video, obtaining a reduced feature vector are performed transforming frames of the video.

6. A method as in claim 1,

wherein each reduced feature vector corresponding to a frame of the training video segment includes features representing chromatic components of the frame and features representing luminance components of the frame.

7. A method as in claim 6,

wherein each reduced feature vector includes fewer features representing chromatic components than features representing luminance components.

8. A method as in claim 1,

wherein each reduced feature vector corresponding to a frame of the training video segment includes features representing red components of the frame, features representing green components of the frame, and features representing blue components of the frame.

9. A method as in claim 3,

wherein the step of segmenting the video into similar and non-similar segments based upon the similarity scores is performed by comparing the similarity scores to an interactively defined similarity threshold.

10. A computer system, comprising:

a processor;

a user interface; and

a processor readable storage medium having processor readable program code embodied on said processor readable storage medium, said processor readable program code for programming the computer system to perform a method of performing a similarity search of a video, the method comprising the steps of;

interactively defining a training video segment from the video;

obtaining reduced feature vectors corresponding to frames of the training video segment; and

training a statistical model using the reduced feature vectors.

11. A computer system, comprising:

a display;

a user interface;

a processor; and

a processor readable storage medium having processor readable program code embodied on said processor readable storage medium, said processor readable program code for programming the computer system to perform a method of presenting a video within a video browser, comprising the steps of;

providing a display window for viewing the video;

displaying a time bar within the video browser, wherein position within the time bar linearly corresponds to elapsed time from a beginning of the video;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each frame in the video to the training video segment using shades of the time bar at positions corresponding to each frame to indicate the similarity measure.

12. A computer system, comprising:

a display;

a user interface;

a processor; and

a processor readable storage medium having processor readable program code embodied on said processor readable storage medium, said processor readable program code for programming the computer system to perform a method of presenting a video within a web-based interface, comprising the steps of;

displaying periodic frames of the video separated by a predetermined time interval;

receiving user training input indicating one or more training video segments from the video; and

displaying a similarity measure of each displayed periodic frame in the video to the training video segment using shades surrounding each displayed periodic frame to indicate the similarity measure.