System and method for processing visual information for event detection

• US 9,256,955 B2
• Filed: 03/13/2013
• Issued: 02/09/2016
• Est. Priority Date: 11/06/2012
• Status: Active Grant

• Current Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Original Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Inventors: O'Gorman, Lawrence, Ho, Tin K., Yin, Yafeng
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US14/416,195
• Publication Number:
  US 20150235379A1
• Time in Patent Office: 1,063 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

A system and method processes visual information including at least one object in motion. The visual information is processed by locating at least one spatial edge of the object, generating a plurality of spatio-temporal gradients for the at least one spatial edge over N frames, and then generating motion blur images from the spatio-temporal gradients. A regression analysis is performed on the motion blur images to determine direction of motion information of the object, and scene activity vectors are then generated for the N frames based on the direction of motion information. An event is detected in the visual information based on the scene activity vectors.

Claims

1. A method for processing information, comprising:

receiving visual information including at least one object in motion;

locating at least one spatial edge of the object in the visual information;

generating a plurality of spatio-temporal gradients for the at least one spatial edge over N frames, where N≧

3;

generating motion blur images from the spatio-temporal gradients;

performing a regression analysis on the motion blur images to determine direction of motion information of the object throughout the N frames;

generating scene activity vectors for the N frames based on the direction of motion information; and

detecting an event in the visual information based on the scene activity vectors generated for the N frames.

2. The method of claim 1, wherein the generating the spatio-temporal gradients includes:

obtaining a spatial edge image for each of the N frames, and

determining the spatio-temporal gradients based on a corresponding one of the spatial edge images and a moving average.

3. The method of claim 2, wherein the moving average is based on a background edge image.

4. The method of claim 2, wherein a region including the at least one spatial edge is smaller than a size of the N frames.

5. The method of claim 1, wherein a location of the object changes in one or more of the N frames.

6. The method of claim 1, wherein the generating the scene activity vectors for the N frames includes:

dividing each of the N frames into blocks;

determining feature parameters for each block;

generating a scene activity vector for each block; and

determining scene activity vectors for the N frames based on the scene activity vectors generated for each block of the N frames.

7. The method of claim 6, wherein the feature parameters include one or more of:

a first parameter based on a sum of non-zero direction values in a corresponding one of the blocks;

a second parameter based on a dominant direction in a corresponding one of the blocks;

a third parameter based on a dominant color in a corresponding one of the blocks;

a fourth parameter based an average velocity of movement in a corresponding one of the blocks;

a fifth parameter based on a degree of consistency or uniformity of motion directions of the object in a corresponding one of the blocks;

a sixth parameter based on density;

or

a seventh parameter based on frequency in time.

8. The method of claim 7, wherein the second parameter is based on the motion vector information for a corresponding one of the blocks.

9. The method of claim 1, wherein the detecting an event includes:

identifying changes in the scene activity vectors the N frames; and

determining that an event has occurred when the changes in scene activity vectors has exceeded a threshold for at least K frames.

10. The method of claim 1, further comprising:

detecting a termination of the event when the scene activity vectors for M frames is less than the threshold, where M≦

N.

11. The method of claim 1, wherein

the object is one or more persons, and

the event corresponds to movement of the one or more persons.

12. The method of claim 1, wherein

the object is traffic, and

the event corresponds to a type of traffic flow.

13. The method of claim 1, wherein

the object is an object in a game, and

the event corresponds to an activity in the game.

21. A non-transitory computer readable medium storing an executable set of instruction, which upon execution configure a processing apparatus to perform the method of claim 1.

14. An apparatus for processing information, comprising:

a storage area configured to store a control program; and

a controller configured to process visual information based on the control program, the controller configured to process the visual information by

locating at least one spatial edge of an object in visual information;

generating a plurality of spatio-temporal gradients for the at least one spatial edge over N frames, where N≧

3;

generating motion blur images from the spatio-temporal gradients;

performing a regression analysis to determine direction of motion information of the object throughout the N frames;

generating scene activity vectors for the N frames based on the direction of motion information; and

detecting an event in the visual information based on the scene activity vectors generated for the N frames.

15. The apparatus of claim 14, wherein the controller is configured to generate the spatio-temporal gradients by:

obtaining a spatial edge image for each of the N frames, and

determining the spatio-temporal gradients based on a corresponding one of the spatial edge images and a moving average.

16. The apparatus of claim 14, wherein the controller is configured to generate the scene activity vectors for the N frames by:

dividing each of the N frames into blocks;

determining feature parameters for each block;

generating a scene activity vector for each block; and

determining scene activity vectors for the N frames based on the scene activity vectors generated for each block of the N frames.

17. The apparatus of claim 16, wherein the feature parameters include one or more of:

a first parameter based on a sum of non-zero direction values in a corresponding one of the blocks;

a second parameter based on a dominant direction in a corresponding one of the blocks;

a third parameter based on a dominant color in a corresponding one of the blocks;

a fourth parameter based an average velocity of movement in a corresponding one of the blocks;

a fifth parameter based on a degree of consistency or uniformity of motion directions of the object in a corresponding one of the blocks;

a sixth parameter based on density;

or

a seventh parameter based on frequency in time.

18. The apparatus of claim 14, wherein the controller is configured to detect the event in the visual information by:

identifying changes in the scene activity vectors the N frames; and

determining that an event has occurred when the changes in scene activity vectors has exceeded a threshold for at least K frames.

19. A system, comprising:

the apparatus according to claim 16;

one or more video sources configured to supply the visual information;

one or more feedback devices configured to generate feedback information; and

a feedback generator interface configured to present output from the processing apparatus, and configured to control the feedback devices to generate the feedback information.

20. The system of claim 19, wherein the feedback device is at least one of an audio presentation device and a video presentation device.