Activity descriptor for video sequences

• US 7,003,038 B2
• Filed: 08/13/2002
• Issued: 02/21/2006
• Est. Priority Date: 09/27/1999
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Sun, Huifang, Kim, Hae-Kwang, Sun, Xinding, Manjunath, Bangalore S., Divakaran, Ajay, Park, Chul-Soo, Vasudevan, Vinod V., Shin, Hyundoo, Rattinassababady, Ganesh, Jesudoss, Manoranjan D.
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US10/217,918
• Publication Number:
  US 20030026340A1
• Time in Patent Office: 1,288 Days
• Field of Search: 375/240.15, 375/240.16, 375/240.17, 375/240.18, 709/247, 708/203, 346/07
• US Class Current: 375/240.16
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A method describes activity in a video sequence. The method measures intensity, direction, spatial, and temporal attributes in the video sequence, and the measured attributes are combined in a digital descriptor of the activity of the video sequence.

Claims

1. A method for describing activity in a compressed video sequence, comprising the steps of:

measuring intensity attributes of the activity in the compressed video sequence;

measuring direction attributes of the activity in the compressed video sequence;

measuring spatial attributes of the activity in the compressed video sequence;

measuring temporal attributes of the activity in the compressed video sequence; and

combining the measured intensity, direction, spatial and temporal attributes of the activity in a digital descriptor of the activity of the compressed video sequence.

2. The method of claim 1 wherein the compressed video sequence includes video components.

3. The method of claim 1 wherein the compressed video sequence includes audio components.

4. The method of claim 1 wherein the compressed video sequence includes a plurality of frames.

5. The method of claim 1 wherein the compressed video sequence includes a plurality of video objects.

6. The method of claim 1 further comprising:

measuring motion vectors representing the activity in the compressed video.

7. The method of claim 6 further comprising:

measuring magnitudes of the motion vectors; and

averaging the measured magnitudes.

8. The method of claim 6 wherein run-lengths of the motion vectors are measured.

9. The method of claim 8 wherein the run-lengths are classified as short, medium or long.

10. The method of claim 9 wherein the compressed video sequence includes a plurality of frames and run-lengths express the number, size, and shape of moving objects in the frames and distributions of the objects across the frames.

11. The method of claim 1 wherein the spatial attributes are expressed as an angle matrix.

12. The method of claim 1 wherein the compressed video sequence include a plurality of frames and each frame includes a plurality of macro-blocks, and further comprising the step of:

averaging the measured attributes over the macro-blocks of the compressed video sequence.

13. The method of claim 1 wherein the compressed video sequence includes a plurality of frames and the attributes are measured for each frame, and further comprising the step of:

randomly selecting one frame as representing the activity of the compressed video sequence.

14. The method of claim 9 wherein the compressed video sequence includes a plurality of frames, and each frame includes a plurality of macro-blocks, and further comprising the steps of:

counting the total number of macro blocks;

counting the number of macro-blocks having with a motion vector (0,0); and

expressing the intensity attribute as a ratio of the two counts.

15. The method of claim 1 wherein the compressed video sequence is one of a plurality of compressed video sequences and each of the other of the plurality of compressed video sequences has a respective digital descriptor of the activity of the one compressed video sequence, and further comprising the step of:

matching the one compressed video sequence with another of the plurality of compressed video sequences based on the digital descriptors.

16. A method for describing an activity represented in a compressed video sequence, comprising:

deriving motion information of the activity representing the compressed video sequence;

determining an intensity attribute corresponding to an intensity level of the represented activity, a direction attribute corresponding to a direction of the represented activity, a spatial attribute corresponding to a spatial dimension of the represented activity, and a temporal attribute corresponding to a temporal dimension of the represented activity, based on the derived motion information; and

describing the activity with a descriptor corresponding to the determined attributes.

17. A method according to claim 16, further comprising:

combining the intensity, the direction, the spatial and the temporal attributes to form the descriptor.

18. The method of claim 16, wherein the motion information is derived by extracting information from a motion vector included in the compressed video sequence.

19. The method of claim 18, wherein:

the extracted information includes one of a magnitude and an angle of the motion vector; and

the motion vector represents one of a predictive frame and a predictive object included in the compressed video sequence.

20. The method of claim 19, wherein:

the compressed video sequence includes a plurality of macro-blocks of pixel data associated with the predictive frame or the predictive object included in the compressed video sequence;

each of the plurality of macro-blocks has an associated attribute; and

at least one of the determined attributes includes the attribute associated with one of the plurality of macro-blocks which is closest to an average of the attributes associated with the plurality of macro-blocks.

21. The method of claim 19, wherein:

the compressed video sequence includes a plurality of macro-blocks of pixel data associated with the predictive frame or the predictive object included in the compressed video sequence; and

the intensity attribute is determined by computing a ratio of the macro-blocks without a motion vector and the total number of macro-blocks associated with the predictive frame or the predictive object.