Methods and interfaces for visualizing activity across video frames in an action keyframe

• US 7,623,677 B2
• Filed: 01/03/2006
• Issued: 11/24/2009
• Est. Priority Date: 06/17/2005
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Girgensohn, Andreas, Kimber, Donald G., Wilcox, Lynn, Shipman, Frank M.
• Primary Examiner(s): Tabatabai; Abolfazl
• Application Number:
  US11/324,355
• Publication Number:
  US 20060284976A1
• Time in Patent Office: 1,421 Days
• Field of Search: 382/103, 382/107, 382/225, 348/135, 348/701, 348/430.1, 348/456, 348/490, 345/440, 345/546, 345/951, 704/236, 704/255, 704/270, 707/2, 707/3, 707/4, 707/104.1, 715/719, 715/839
• US Class Current: 382/103
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion

Abstract

Techniques for generating action keyframes for a fixed-position camera based on the identification of activity in the video, an assessment of the importance of the activity, object recognition in the video, and interaction techniques for seeing more details of the video are presented. In different embodiments of the invention, the importance of activity is determined based on the amount of activity, important locations in the video streams, detected features such as faces, and events from other sensors.

Claims

1. A computer implemented method of, on a computer system having one or more processors and memory storing programs executed by the one or more processors, visualizing an activity represented in a video segment by manipulating a sequence of video frames from the video segment comprising the steps of:

(a) classifying a plurality of pixels in the sequence of video frames as representing a foreground object or a background object;

(b) assigning one or more alpha values for each classified pixel in each video frame, based at least in part on the classification of the classified pixel;

(c) manipulating the plurality of alpha values in order to emphasize one or more of the pixels classified as representing a foreground object; and

(d) applying the plurality of alpha values to each pixel to visualize the activity in the video segment within a single still image.

2. The method of claim 1, where the video frames are part of the video stream and step (a) further comprises the steps of selecting an optimal sampling rate to select the plurality of video frames and thereby classify the plurality of pixels as representing a foreground object or a background object.

3. The method of claim 2, where in step (a) the foreground pixels and the background pixels are classified by determining if there is change above a threshold opacity in a pixel between the plurality of video frames at the selected sampling rate.

4. The method of claim 2, where in step (a) the foreground pixels and the background pixels are classified by setting a threshold luminance to determine if there is change above a threshold opacity in a pixel between the plurality of video frames at the selected sample rate.

5. The method of claim 4, wherein the threshold luminance is between:

about 4%; and

about 6%.

6. The method of claim 2, wherein the optimal sampling rate is selected from the group consisting of a fixed sampling rate and a variable sampling rate.

7. The method of claim 2, wherein the sampling rate is between:

about 0.5 frames per second; and

about 2 frames per second.

8. The method of claim 1, where in step (b) an alpha value is computed for foreground object pixels and a separate alpha value is computed for background pixels.

16. The method of claim 1, where in step (b) the alpha value opacity is increased.

9. A computer implemented method of, on a computer system having one or more processors and memory storing programs executed by the one or more processors, visualizing an activity corresponding to an event in one or more frames comprising the steps of:

(a) classifying a plurality of pixels in the plurality of frames as representing a foreground object or a background object;

(b) assigning one or more alpha values for each classified pixel in each frame;

(c) manipulating the plurality of alpha values by applying one or more of the functions selected from the group consisting of accumulating an average weighted by the alpha value for each pixel across a specified number of frames, normalizing the alpha values for each pixel in each frame of the video stream and smoothing the alpha mask applied to each pixel; and

(d) applying the plurality of alpha values to each pixel to visualize the activity.

10. The method of claim 9, wherein the function of smoothing the alpha value involves applying a separate smoothed alpha value to foreground pixels.

11. The method of claim 10, wherein to emphasize an activity or region within a sample, the smoothed alpha value variations are selected from the group consisting of variation across the sample and variation within the sample.

12. The method of claim 10, where in step (c) foreground pixels are tinted with a translucent color;

whereby the color can be varied over time to indicate temporal order.

13. The method of claim 10, where in step (c) background pixels are tinted with a translucent color;

whereby the color applied to the background pixel can be varied based on a distance of the pixel from a nearest pixel that was ever part of the foreground.

14. The method of claim 10, where in step (c) a saturation of the background pixels towards grayscale is reduced by mixing their color with a luminance of their color;

whereby a mixing color applied to the background pixel can be varied based on a distance of the pixel from a nearest pixel that was ever part of the foreground.

15. The method of claim 10, wherein a colored halo is drawn around a shape created by the foreground pixels.

17. The method of claim 9, further comprising:

(e) generating one or more keyframes of one or more events in the video stream;

wherein the keyframes further comprise an alpha blend of objects from different times to visualize the activity.

18. The method of claim 17, wherein the keyframes further comprise showing object trajectories.

19. The method of claim 17, wherein a user can highlight a shape and track of a position of one or more interesting features in the video by clicking on one or more shapes in a single video frame or in a keyframe.

20. The method of claim 17, wherein a color and/or transparency of different objects can be altered to emphasize activity.

21. The method of claim 17, wherein a trajectory of different objects are highlighted in separate keyframes.

22. The method of claim 17, wherein an activity observed in one keyframe is used to identify an object;

wherein the object is further identified in other keyframes based on time.

23. A machine readable storage medium including a computer program of instructions executable by a computer to visualize an activity represented in a video segment by manipulating a sequence of video frames from the video segment, the instructions including instructions for:

(a) classifying a plurality of pixels in the sequence of video frames as representing a foreground object or a background object;

(b) assigning one or more alpha values for each classified pixel in each video frame, based at least in part on the classification of the classified pixel;

(c) manipulating the plurality of alpha values in order to emphasize one or more of the pixels classified as representing a foreground object; and

(d) applying the plurality of alpha values to each pixel to visualize the activity in the video segment within a single still image.

24. A system or apparatus for visualizing an activity represented in a video segment by manipulating a sequence of video frames from the video segment, comprising:

memory

one or more processors; and

one or more modules stored in the memory and executed by the one or more processors, the one or more modules including instructions to;

(a) classify a plurality of pixels in the sequence of video frames as representing a foreground objects or a background object;

(b) assign one or more alpha values for each classified pixel in each video frame, based at least in part on the classification of the classified pixel;

(c) manipulate the plurality of alpha values in order to emphasize one or more of the pixels classified as representing a foreground object; and

(d) apply the plurality of alpha values to each pixel to visualize the activity in the video segment within a single still image.

25. A machine readable storage medium including a computer program of instructions executable by a computer to visualize an activity corresponding to an event in one or more frames, the instructions including instructions for:

(a) classifying a plurality of pixels in the plurality of frames as representing a foreground object or a background object;

(b) assigning one or more alpha values for each classified pixel in each frame;

(c) manipulating the plurality of alpha values by applying one or more of the functions selected from the group consisting of accumulating an average weighted by the alpha value for each pixel across a specified number of frames, normalizing the alpha values for each pixel in each frame of the video stream and smoothing the alpha mask applied to each pixel; and

(d) applying the plurality of alpha values to each pixel to visualize the activity.

26. A system or apparatus for visualizing an activity corresponding to an event in one or more frames, comprising:

memory

one or more processors; and

one or more modules stored in the memory and executed by the one or more processors, the one or more modules including instructions to;

(a) classify a plurality of pixels in the plurality of frames as representing a foreground object or a background object;

(b) assign one or more alpha values for each classified pixel in each frame;

(c) manipulate the plurality of alpha values by applying one or more of the functions selected from the group consisting of accumulating an average weighted by the alpha value for each pixel across a specified number of frames, normalizing the alpha values for each pixel in each frame of the video stream and smoothing the alpha mask applied to each pixel; and

(d) apply the plurality of alpha values to each pixel to visualize the activity.