Content-based characterization of video frame sequences

• US 20050031038A1
• Filed: 09/02/2004
• Published: 02/10/2005
• Est. Priority Date: 09/25/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Ma, Yufei
• Granted Patent: US 9,779,303 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A system and process for video characterization that facilitates video classification and retrieval, as well as motion detection, applications. This involves characterizing a video sequence with a gray scale image having pixel levels that reflect the intensity of motion associated with a corresponding region in the sequence of video frames. The intensity of motion is defined using any of three characterizing processes. Namely, a perceived motion energy spectrum (PMES) characterizing process that represents object-based motion intensity over the sequence of frames, a spatio-temporal entropy (STE) characterizing process that represents the intensity of motion based on color variation at each pixel location, a motion vector angle entropy (MVAE) characterizing process which represents the intensity of motion based on the variation of motion vector angles.

Claims

1. A computer-implemented process for characterizing a sequence of video frames, comprising using a computer to perform the following process actions:

deriving from the sequence of video frames a separate value indicative of the intensity of the motion depicted over the sequence in each of a plurality of frame regions;

generating an image wherein each pixel of the image has a level reflecting the value indicating the intensity of motion, relative to all such values, associated with the region containing the corresponding pixel location.

2. The process of claim 1, wherein each region used to partition each frame is macro block sized.

3. The process of claim 1, wherein each region used to partition each frame is pixel sized.

4-18. -18. (cancelled)

19. A system for finding one or more video shots in a database, each of which comprises a sequence of video frames which depict motion similar to that specified by a user in a user query, comprising:

a general purpose computing device;

the database which is accessible by the computing device and which comprises,

a plurality of characterizing images each of which represents a shot, wherein each characterizing image is an image comprising pixels reflecting the intensity of motion associated with a corresponding region in the sequence of video frames;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input the user query which comprises a characterizing image that characterizes motion in the same manner as at least some of the characterizing images contained in the database, and

compare the user query image to characterizing images contained in the database that characterize motion in the same manner as the user query image to find characterizing images that exhibit a degree of similarity equaling or exceeding a prescribed minimum similarity threshold.

20. The system of claim 19, further comprising a program module for providing information to the user for accessing the shot corresponding to at least one of any characterizing images contained in the database that were found to exhibit a degree of similarity equaling or exceeding the prescribed minimum similarity threshold.

21. The system of claim 19, wherein the database further comprises a plurality of pointers each of which identifies the location where the shot corresponding to one of the characterizing images contained in the database is stored, and wherein the system further comprises program modules for:

accessing the shot corresponding to at least one of any characterizing images contained in the database that were found to exhibit a degree of similarity equaling or exceeding the prescribed minimum similarity threshold, and

providing each accessed shot to the user.

22. The system of claim 19, wherein the characterizing image associated with the user'"'"'s query comprises a characterizing image that was generated in the same manner as at least some of the characterizing images contained in the database.

23. The system of claim 19, wherein the characterizing image associated with the user'"'"'s query comprises a characterizing image created by the user to simulate the manner in which motion is represented in at least some of the characterizing images contained in the database.

24-37. -37. (cancelled)

38. A computer-readable medium having computer-executable instructions for characterizing at least a portion of a shot that is made up of a sequence of video frames, said computer-executable instructions comprising:

deriving from the sequence of video frames a separate value indicative of the intensity of the motion depicted over the sequence in each of a plurality of frame regions;

generating a gray scale image comprising pixels which each have a gray scale level reflecting the intensity of motion associated with a corresponding region in the sequence of video frames.

40. The computer-readable medium of claim 38, wherein the frames of the shot comprise pixel color data, and wherein the instruction for deriving a value indicative of the intensity of the motion for each frame region, comprises sub-modules for:

inputting a prescribed number of frames in shot sequence order;

extracting and storing pixel color data from the inputted frames;

computing a spatio-temporal energy (STE) value for each pixel location.

41. The computer-readable medium of claim 38, wherein at least some of the frames of the shot comprise motion vector data, and wherein the instruction for deriving a value indicative of the intensity of the motion for each frame region, comprises sub-modules for:

inputting a prescribed number of frames, which comprise motion vector data, in shot sequence order;

extracting and storing the motion vector data from the inputted frames;

computing a motion vector angle energy value for each frame region.

39. (cancelled)

42-46. -46. (cancelled)